Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.777704914s trace: https://jaeger.dev.storj.io/trace/747a47aa26cdc65f FAIL (2025-11-24 05:25:32 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.777736965s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.010160606s trace: https://jaeger.dev.storj.io/trace/68b6311f086e60c6 FAIL (2025-11-24 05:25:08 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.010188409s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................X...X.........XX..X..] duration: 5.903807126s trace: https://jaeger.dev.storj.io/trace/599c0302b0cfc801 FAIL (2025-11-24 05:24:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 24 05:23:02.449603695 (357h19m1.001074156s) [upload=314521]: Starting upload Nov 24 05:23:02.449706301 (357h19m1.001176762s) [upload=314521]: Done starting upload Nov 24 05:23:02.449961758 (357h19m1.001432219s) [upload=314521]: Uploading segments... Nov 24 05:23:02.450259785 (357h19m1.001730256s) [upload=314521]: Got next segment. Inline: false Nov 24 05:23:02.450324648 (357h19m1.001795109s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:23:02.450332814 (357h19m1.001803285s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:23:02.704332160 (357h19m1.255802621s) [seg_pos={0 0}, upload=314521]: Begin upload segment... Nov 24 05:23:02.704394768 (357h19m1.255865239s) [seg_pos={0 0}, upload=314521]: Done begin upload segment. Nov 24 05:23:02.704436909 (357h19m1.25590736s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.704601662 (357h19m1.256072133s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705249546 (357h19m1.256720017s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705593661 (357h19m1.257064122s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705681198 (357h19m1.257151659s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706230524 (357h19m1.257700985s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706359789 (357h19m1.25783025s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707030456 (357h19m1.258500917s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707439435 (357h19m1.258909896s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707643253 (357h19m1.259113725s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707916354 (357h19m1.259386825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708201466 (357h19m1.259671937s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708344449 (357h19m1.25981491s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708793404 (357h19m1.260263865s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708834722 (357h19m1.260305183s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.710680426 (357h19m1.262150877s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:04.472015490 (357h19m3.023485951s) [upload=314521]: Next returned nil segment Nov 24 05:23:04.472034676 (357h19m3.023505137s) [upload=314521]: Waiting for error group managing segments... Nov 24 05:23:04.788643757 (357h19m3.340114218s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.883924979 (357h19m3.43539544s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.904748416 (357h19m3.456218877s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.905616579 (357h19m3.45708704s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.911750517 (357h19m3.463220978s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.920810467 (357h19m3.472280928s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.924497988 (357h19m3.47596846s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.929369955 (357h19m3.480840416s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.956308374 (357h19m3.507778835s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.982610122 (357h19m3.534080603s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.367340593 (357h19m3.918811064s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.593064333 (357h19m4.144534804s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661884283 (357h19m4.213354744s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661905744 (357h19m4.213376195s) [seg_pos={0 0}, upload=314521]: Segment reached optimal threshold of 13 pieces. Nov 24 05:23:05.662121505 (357h19m4.213591966s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662133698 (357h19m4.213604149s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662158315 (357h19m4.213628776s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection Nov 24 05:23:05.662169947 (357h19m4.213640408s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662326175 (357h19m4.213796626s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662336364 (357h19m4.213806825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662363365 (357h19m4.213833826s) [seg_pos={0 0}, upload=314521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:23:05.662470149 (357h19m4.21394061s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:23:05.662478434 (357h19m4.213948895s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:23:05.964002134 (357h19m4.515472585s) [upload=314521]: Done uploading segments... Nov 24 05:23:05.964027141 (357h19m4.515497613s) [upload=314521]: Upload finished. err: <nil> Duration: 5.90384084s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 1.991306829s trace: https://jaeger.dev.storj.io/trace/68b6311f086e60c6 FAIL (2025-11-24 05:24:31 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 1.991332489s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.947784208s trace: https://jaeger.dev.storj.io/trace/747a47aa26cdc65f FAIL (2025-11-24 05:24:19 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.947816371s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........................................X...X.........XX..X.] duration: 21.065973096s trace: https://jaeger.dev.storj.io/trace/599c0302b0cfc801 FAIL (2025-11-24 05:23:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 24 05:23:02.449603695 (357h19m1.001074156s) [upload=314521]: Starting upload Nov 24 05:23:02.449706301 (357h19m1.001176762s) [upload=314521]: Done starting upload Nov 24 05:23:02.449961758 (357h19m1.001432219s) [upload=314521]: Uploading segments... Nov 24 05:23:02.450259785 (357h19m1.001730256s) [upload=314521]: Got next segment. Inline: false Nov 24 05:23:02.450324648 (357h19m1.001795109s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:23:02.450332814 (357h19m1.001803285s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:23:02.704332160 (357h19m1.255802621s) [seg_pos={0 0}, upload=314521]: Begin upload segment... Nov 24 05:23:02.704394768 (357h19m1.255865239s) [seg_pos={0 0}, upload=314521]: Done begin upload segment. Nov 24 05:23:02.704436909 (357h19m1.25590736s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.704601662 (357h19m1.256072133s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705249546 (357h19m1.256720017s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705593661 (357h19m1.257064122s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705681198 (357h19m1.257151659s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706230524 (357h19m1.257700985s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706359789 (357h19m1.25783025s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707030456 (357h19m1.258500917s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707439435 (357h19m1.258909896s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707643253 (357h19m1.259113725s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707916354 (357h19m1.259386825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708201466 (357h19m1.259671937s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708344449 (357h19m1.25981491s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708793404 (357h19m1.260263865s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708834722 (357h19m1.260305183s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.710680426 (357h19m1.262150877s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:04.472015490 (357h19m3.023485951s) [upload=314521]: Next returned nil segment Nov 24 05:23:04.472034676 (357h19m3.023505137s) [upload=314521]: Waiting for error group managing segments... Nov 24 05:23:04.788643757 (357h19m3.340114218s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.883924979 (357h19m3.43539544s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.904748416 (357h19m3.456218877s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.905616579 (357h19m3.45708704s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.911750517 (357h19m3.463220978s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.920810467 (357h19m3.472280928s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.924497988 (357h19m3.47596846s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.929369955 (357h19m3.480840416s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.956308374 (357h19m3.507778835s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.982610122 (357h19m3.534080603s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.367340593 (357h19m3.918811064s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.593064333 (357h19m4.144534804s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661884283 (357h19m4.213354744s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661905744 (357h19m4.213376195s) [seg_pos={0 0}, upload=314521]: Segment reached optimal threshold of 13 pieces. Nov 24 05:23:05.662121505 (357h19m4.213591966s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662133698 (357h19m4.213604149s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662158315 (357h19m4.213628776s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection Nov 24 05:23:05.662169947 (357h19m4.213640408s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662326175 (357h19m4.213796626s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662336364 (357h19m4.213806825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662363365 (357h19m4.213833826s) [seg_pos={0 0}, upload=314521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:23:05.662470149 (357h19m4.21394061s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:23:05.662478434 (357h19m4.213948895s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:23:05.964002134 (357h19m4.515472585s) [upload=314521]: Done uploading segments... Nov 24 05:23:05.964027141 (357h19m4.515497613s) [upload=314521]: Upload finished. err: <nil> Duration: 21.065994146s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 1.982896015s trace: https://jaeger.dev.storj.io/trace/68b6311f086e60c6 FAIL (2025-11-24 05:23:53 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 1.982917026s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........................................X...X.........XX..X] duration: 30.003361393s trace: https://jaeger.dev.storj.io/trace/599c0302b0cfc801 FAIL (2025-11-24 05:23:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 24 05:23:02.449603695 (357h19m1.001074156s) [upload=314521]: Starting upload Nov 24 05:23:02.449706301 (357h19m1.001176762s) [upload=314521]: Done starting upload Nov 24 05:23:02.449961758 (357h19m1.001432219s) [upload=314521]: Uploading segments... Nov 24 05:23:02.450259785 (357h19m1.001730256s) [upload=314521]: Got next segment. Inline: false Nov 24 05:23:02.450324648 (357h19m1.001795109s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:23:02.450332814 (357h19m1.001803285s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:23:02.704332160 (357h19m1.255802621s) [seg_pos={0 0}, upload=314521]: Begin upload segment... Nov 24 05:23:02.704394768 (357h19m1.255865239s) [seg_pos={0 0}, upload=314521]: Done begin upload segment. Nov 24 05:23:02.704436909 (357h19m1.25590736s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.704601662 (357h19m1.256072133s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705249546 (357h19m1.256720017s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705593661 (357h19m1.257064122s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.705681198 (357h19m1.257151659s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706230524 (357h19m1.257700985s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.706359789 (357h19m1.25783025s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707030456 (357h19m1.258500917s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707439435 (357h19m1.258909896s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707643253 (357h19m1.259113725s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.707916354 (357h19m1.259386825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708201466 (357h19m1.259671937s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708344449 (357h19m1.25981491s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708793404 (357h19m1.260263865s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.708834722 (357h19m1.260305183s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:02.710680426 (357h19m1.262150877s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Uploading piece... Nov 24 05:23:04.472015490 (357h19m3.023485951s) [upload=314521]: Next returned nil segment Nov 24 05:23:04.472034676 (357h19m3.023505137s) [upload=314521]: Waiting for error group managing segments... Nov 24 05:23:04.788643757 (357h19m3.340114218s) [noise=true , address=77.90.31.15:4697 , piece_id=M7IXD6EXW5CJMEHHFTWIKL3BOTGTO46FHXMG4VSYUA2JSAVNNOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.883924979 (357h19m3.43539544s) [noise=true , address=45.91.116.9:4263 , piece_id=X7DXQMI7BARFP3HXG37RDMMEZX2UENQ3V5PMZVHQP3QPB22UOOUQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.904748416 (357h19m3.456218877s) [noise=true , address=145.79.227.24:4583 , piece_id=AQ4RPI4YSZ4RWG7LZJ446BQE4F3JLE7EV5KMBT5NAOLIPBOTTKVA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.905616579 (357h19m3.45708704s) [noise=true , address=145.79.227.21:4474 , piece_id=QZJG7QREDA4YB7L3Z5LRZO4JEGSON2EDMWCXERPYVYUQDOUN7ZPA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.911750517 (357h19m3.463220978s) [noise=true , address=77.90.31.13:4626 , piece_id=WCO3L3P7VYS3NUL2DHZUDC3NLEMSP7KYTS7VG7I6PGDA2YLWCSCA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.920810467 (357h19m3.472280928s) [noise=true , address=145.79.227.14:4838 , piece_id=OWWFG2DWVP55FWHD3CLG3PJEBBEIDFFO3R3EBRW7P27RU3OF53OQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.924497988 (357h19m3.47596846s) [noise=true , address=145.79.227.17:4971 , piece_id=P43EREHGSJCMHDC5ZULWYHNPW22SIGWOTEYYGCWV3LYRZM34M4CQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.929369955 (357h19m3.480840416s) [noise=true , address=77.90.31.14:4661 , piece_id=CNCRYTPG75Z6HK5CXSNI3NKDOGJVF5NECN5UZTT2VPMJ56EPYNJQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.956308374 (357h19m3.507778835s) [noise=true , address=145.79.227.8:4331 , piece_id=CT3RJJ73AU7M7XAJK3DC54H4EXHIJLKU6OXCR7MCMYEBWKKSWE5Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:04.982610122 (357h19m3.534080603s) [noise=true , address=77.90.31.21:4879 , piece_id=G5O2UHPQZ75I63YCRYAJEWMO7HSCW2RH3QVHM4DUD37A5SOVBXYQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.367340593 (357h19m3.918811064s) [noise=true , address=145.79.227.9:4365 , piece_id=J4F4MS5R3IRZRVFHDM6V2WGGFG4VW62DRMS6D4GWGW5CNFI35AMA, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.593064333 (357h19m4.144534804s) [noise=true , address=145.79.227.31:4797 , piece_id=EDWIDLR7KLDQNG6BKWFTXOUOJLKJKSM7ILTINHY5F3KYLTO3FDBQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661884283 (357h19m4.213354744s) [noise=true , address=145.79.227.24:4582 , piece_id=OCZFWWWK2ECNRISYQK5ILWJZSDDPZKWK7HRBGPNBMDHTA4OJDQQQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: <nil> Nov 24 05:23:05.661905744 (357h19m4.213376195s) [seg_pos={0 0}, upload=314521]: Segment reached optimal threshold of 13 pieces. Nov 24 05:23:05.662121505 (357h19m4.213591966s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662133698 (357h19m4.213604149s) [noise=true , address=145.79.227.26:4653 , piece_id=ZK63CKT3NLXRYREB2QXKUL7HLWVIODAEEIKSUQCHQTIO2DXICB6Q, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662158315 (357h19m4.213628776s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37484->145.79.227.16:4909: use of closed network connection Nov 24 05:23:05.662169947 (357h19m4.213640408s) [noise=true , address=145.79.227.16:4909 , piece_id=YM5K4ALDMGNRRMOVAVIRVS2TC7IAG5KEJW2BZUZNCS4XHSWR7YCQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662326175 (357h19m4.213796626s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Nov 24 05:23:05.662336364 (357h19m4.213806825s) [noise=true , address=145.79.227.28:4727 , piece_id=MKLU2V2DXZXGKJDCYRAVN42OHIQK4ZCNHOUG4MNW2JMTJP6UK5YQ, seg_pos={0 0}, upload=314521]: Long tail cancellation due to optimal threshold Nov 24 05:23:05.662363365 (357h19m4.213833826s) [seg_pos={0 0}, upload=314521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:23:05.662470149 (357h19m4.21394061s) [upload=314521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:23:05.662478434 (357h19m4.213948895s) [upload=314521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:23:05.964002134 (357h19m4.515472585s) [upload=314521]: Done uploading segments... Nov 24 05:23:05.964027141 (357h19m4.515497613s) [upload=314521]: Upload finished. err: <nil> Duration: 30.003387771s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.185031632s trace: https://jaeger.dev.storj.io/trace/747a47aa26cdc65f FAIL (2025-11-24 05:23:08 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.185065606s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 1.972677151s trace: https://jaeger.dev.storj.io/trace/68b6311f086e60c6 FAIL (2025-11-24 05:23:04 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 1.972708191s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.027240542s trace: https://jaeger.dev.storj.io/trace/68b6311f086e60c6 FAIL (2025-11-24 05:22:35 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.027269888s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................................X...X.........XX..] duration: 5.685644447s trace: https://jaeger.dev.storj.io/trace/656c5eee210484ef FAIL (2025-11-24 05:22:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4 05:20:27.101694298 (357h16m25.653164749s) [upload=314482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:20:27.101700471 (357h16m25.653170922s) [upload=314482]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:20:30.941084752 (357h16m29.492555354s) [seg_pos={0 0}, upload=314482]: Begin upload segment... Nov 24 05:20:30.941151700 (357h16m29.492622452s) [seg_pos={0 0}, upload=314482]: Done begin upload segment. Nov 24 05:20:30.941192067 (357h16m29.492662528s) [noise=true , address=77.90.31.12:4592 , piece_id=FGZSDMDUHHWA25KVCJ67FC75SAO7TTFOOKTXLMJEVHK3G25UOEJQ, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.942182242 (357h16m29.493652703s) [noise=true , address=145.79.227.24:4581 , piece_id=WGL7V2A7QYIB7W5PIQW6VGFVEXWZBVG6BZJEWRZFCR32NRPAF42A, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.942590840 (357h16m29.494061301s) [noise=true , address=45.91.116.11:4118 , piece_id=SKTEMZIMVA3B2HC2Q3ZJI2JZXDJBI4TQ6HWCQU5SRX7B4Y2URGBQ, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.943065604 (357h16m29.494536065s) [noise=true , address=145.79.227.4:4186 , piece_id=WA6P7W3GAYWQQCN4WTLF2TPZEF4YH333KS3CRAYCSUDFXSETUYJA, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.943513267 (357h16m29.494983718s) [noise=true , address=145.79.227.14:4839 , piece_id=4GCQV2V53YRT542J2H4BONZGE5LC7BV6GJ5NSLEDCNKAR3ZOIMCQ, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.944009952 (357h16m29.495480413s) [noise=true , address=77.90.31.18:4805 , piece_id=ZOPHHHDPANVCR24DYEVFTPTEI7QZWJC2CSKHTA3AIOE4LVSAIO7A, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.944393923 (357h16m29.495864385s) [noise=true , address=145.79.227.15:4874 , piece_id=HX222X63BP3XZJDWRUA66L25RB5OQC6QVO64KFE2YIVAASMH5V5Q, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.944915867 (357h16m29.496386369s) [noise=true , address=145.79.227.27:4690 , piece_id=QY2UEG6CDJQC7PK5Y4CV7NNWAO6OIU7G7AM4LOULWAYIIXP72DYQ, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.945511261 (357h16m29.496981712s) [noise=true , address=145.79.227.13:4804 , piece_id=MY352JGM6TNDPQRCT5NAQ5F43PQZFLCF3LAPBOXKERH4LKVXJMDA, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.946892241 (357h16m29.498362702s) [noise=true , address=145.79.227.31:4799 , piece_id=ORXIPX6OLF6KRZMILCLQLI6DJ4R3WWD2BG7COIFXCC3LTIJAR7CQ, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.948306093 (357h16m29.499776554s) [noise=true , address=145.79.227.9:4365 , piece_id=IVA4Z3LIMJDEZJAOQOMXHHK6YER6IJLIT6T5TYVALG4YKSOIITFA, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.949923172 (357h16m29.501393633s) [noise=true , address=77.90.31.10:4518 , piece_id=QGJDEHZDAG6WIVH2OI27WXBUTPGWJLYW7YUF626EHDXQUDK2GBCQ, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.951243466 (357h16m29.502713928s) [noise=true , address=45.91.116.9:4263 , piece_id=NWOUVIIGEJTZMBFVMGAHSAW2QXOVWQ3R6SVAFWKGZBLPI2CAJE4Q, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.952429815 (357h16m29.503900266s) [noise=true , address=45.91.116.8:4045 , piece_id=WY4ATDPUHVRHYJRQN3LMXJIKCFZIQRGD6DZL5LGZSHFDD2IL565A, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.953561660 (357h16m29.505032121s) [noise=true , address=145.79.227.20:4438 , piece_id=6YUL34NHDGETIJVH4IDTJ7OATWQDHDSS7STNYMIWDXIYDSKMVUHQ, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.953950061 (357h16m29.505420522s) [noise=true , address=145.79.227.18:4006 , piece_id=B3XG2EOIQVHMIDVCUSRTJPLWQFD7LMUKWCCNWKVO3VNOXYCSGUWQ, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:32.649721339 (357h16m31.2011918s) [upload=314482]: Next returned nil segment Nov 24 05:20:32.649739322 (357h16m31.201209804s) [upload=314482]: Waiting for error group managing segments... Nov 24 05:20:32.999561294 (357h16m31.551031755s) [noise=true , address=45.91.116.8:4045 , piece_id=WY4ATDPUHVRHYJRQN3LMXJIKCFZIQRGD6DZL5LGZSHFDD2IL565A, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:33.052276352 (357h16m31.603746803s) [noise=true , address=45.91.116.11:4118 , piece_id=SKTEMZIMVA3B2HC2Q3ZJI2JZXDJBI4TQ6HWCQU5SRX7B4Y2URGBQ, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:33.060891888 (357h16m31.612362339s) [noise=true , address=77.90.31.18:4805 , piece_id=ZOPHHHDPANVCR24DYEVFTPTEI7QZWJC2CSKHTA3AIOE4LVSAIO7A, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:33.096526542 (357h16m31.647997003s) [noise=true , address=45.91.116.9:4263 , piece_id=NWOUVIIGEJTZMBFVMGAHSAW2QXOVWQ3R6SVAFWKGZBLPI2CAJE4Q, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:33.108779213 (357h16m31.660249674s) [noise=true , address=77.90.31.10:4518 , piece_id=QGJDEHZDAG6WIVH2OI27WXBUTPGWJLYW7YUF626EHDXQUDK2GBCQ, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:33.127759234 (357h16m31.679229695s) [noise=true , address=77.90.31.12:4592 , piece_id=FGZSDMDUHHWA25KVCJ67FC75SAO7TTFOOKTXLMJEVHK3G25UOEJQ, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:33.149265876 (357h16m31.700736338s) [noise=true , address=145.79.227.27:4690 , piece_id=QY2UEG6CDJQC7PK5Y4CV7NNWAO6OIU7G7AM4LOULWAYIIXP72DYQ, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:34.189353961 (357h16m32.740824422s) [noise=true , address=145.79.227.24:4581 , piece_id=WGL7V2A7QYIB7W5PIQW6VGFVEXWZBVG6BZJEWRZFCR32NRPAF42A, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:34.317358835 (357h16m32.868829296s) [noise=true , address=145.79.227.9:4365 , piece_id=IVA4Z3LIMJDEZJAOQOMXHHK6YER6IJLIT6T5TYVALG4YKSOIITFA, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:34.801613418 (357h16m33.353083879s) [noise=true , address=145.79.227.20:4438 , piece_id=6YUL34NHDGETIJVH4IDTJ7OATWQDHDSS7STNYMIWDXIYDSKMVUHQ, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:35.533009581 (357h16m34.084480042s) [noise=true , address=145.79.227.31:4799 , piece_id=ORXIPX6OLF6KRZMILCLQLI6DJ4R3WWD2BG7COIFXCC3LTIJAR7CQ, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:35.881405926 (357h16m34.432876387s) [noise=true , address=145.79.227.18:4006 , piece_id=B3XG2EOIQVHMIDVCUSRTJPLWQFD7LMUKWCCNWKVO3VNOXYCSGUWQ, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:36.009138310 (357h16m34.560608772s) [noise=true , address=145.79.227.14:4839 , piece_id=4GCQV2V53YRT542J2H4BONZGE5LC7BV6GJ5NSLEDCNKAR3ZOIMCQ, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:36.009166294 (357h16m34.560636755s) [seg_pos={0 0}, upload=314482]: Segment reached optimal threshold of 13 pieces. Nov 24 05:20:36.009604859 (357h16m34.561075321s) [noise=true , address=145.79.227.15:4874 , piece_id=HX222X63BP3XZJDWRUA66L25RB5OQC6QVO64KFE2YIVAASMH5V5Q, seg_pos={0 0}, upload=314482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: read tcp 188.166.225.170:48954->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 24 05:20:36.009615309 (357h16m34.56108577s) [noise=true , address=145.79.227.15:4874 , piece_id=HX222X63BP3XZJDWRUA66L25RB5OQC6QVO64KFE2YIVAASMH5V5Q, seg_pos={0 0}, upload=314482]: Long tail cancellation due to optimal threshold Nov 24 05:20:36.009966337 (357h16m34.561436798s) [noise=true , address=145.79.227.13:4804 , piece_id=MY352JGM6TNDPQRCT5NAQ5F43PQZFLCF3LAPBOXKERH4LKVXJMDA, seg_pos={0 0}, upload=314482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: write tcp 188.166.225.170:49618->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:49618->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49618->145.79.227.13:4804: use of closed network connection Nov 24 05:20:36.010005041 (357h16m34.561475502s) [noise=true , address=145.79.227.13:4804 , piece_id=MY352JGM6TNDPQRCT5NAQ5F43PQZFLCF3LAPBOXKERH4LKVXJMDA, seg_pos={0 0}, upload=314482]: Long tail cancellation due to optimal threshold Nov 24 05:20:36.010055868 (357h16m34.561526339s) [noise=true , address=145.79.227.4:4186 , piece_id=WA6P7W3GAYWQQCN4WTLF2TPZEF4YH333KS3CRAYCSUDFXSETUYJA, seg_pos={0 0}, upload=314482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: write tcp 188.166.225.170:50504->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:50504->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50504->145.79.227.4:4186: use of closed network connection Nov 24 05:20:36.010069965 (357h16m34.561540426s) [noise=true , address=145.79.227.4:4186 , piece_id=WA6P7W3GAYWQQCN4WTLF2TPZEF4YH333KS3CRAYCSUDFXSETUYJA, seg_pos={0 0}, upload=314482]: Long tail cancellation due to optimal threshold Nov 24 05:20:36.010102216 (357h16m34.561572677s) [seg_pos={0 0}, upload=314482]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:20:36.010201435 (357h16m34.561671896s) [upload=314482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:20:36.010209561 (357h16m34.561680022s) [upload=314482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:20:36.488610929 (357h16m35.04008139s) [upload=314482]: Done uploading segments... Nov 24 05:20:36.488637500 (357h16m35.040107951s) [upload=314482]: Upload finished. err: <nil> Duration: 5.685671819s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.930978944s trace: https://jaeger.dev.storj.io/trace/747a47aa26cdc65f FAIL (2025-11-24 05:22:05 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.931013641s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.015592322s trace: https://jaeger.dev.storj.io/trace/68b6311f086e60c6 FAIL (2025-11-24 05:21:44 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.015624673s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................................X...X.........XX.] duration: 5.870472902s trace: https://jaeger.dev.storj.io/trace/656c5eee210484ef FAIL (2025-11-24 05:21:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4 05:20:27.101694298 (357h16m25.653164749s) [upload=314482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:20:27.101700471 (357h16m25.653170922s) [upload=314482]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:20:30.941084752 (357h16m29.492555354s) [seg_pos={0 0}, upload=314482]: Begin upload segment... Nov 24 05:20:30.941151700 (357h16m29.492622452s) [seg_pos={0 0}, upload=314482]: Done begin upload segment. Nov 24 05:20:30.941192067 (357h16m29.492662528s) [noise=true , address=77.90.31.12:4592 , piece_id=FGZSDMDUHHWA25KVCJ67FC75SAO7TTFOOKTXLMJEVHK3G25UOEJQ, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.942182242 (357h16m29.493652703s) [noise=true , address=145.79.227.24:4581 , piece_id=WGL7V2A7QYIB7W5PIQW6VGFVEXWZBVG6BZJEWRZFCR32NRPAF42A, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.942590840 (357h16m29.494061301s) [noise=true , address=45.91.116.11:4118 , piece_id=SKTEMZIMVA3B2HC2Q3ZJI2JZXDJBI4TQ6HWCQU5SRX7B4Y2URGBQ, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.943065604 (357h16m29.494536065s) [noise=true , address=145.79.227.4:4186 , piece_id=WA6P7W3GAYWQQCN4WTLF2TPZEF4YH333KS3CRAYCSUDFXSETUYJA, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.943513267 (357h16m29.494983718s) [noise=true , address=145.79.227.14:4839 , piece_id=4GCQV2V53YRT542J2H4BONZGE5LC7BV6GJ5NSLEDCNKAR3ZOIMCQ, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.944009952 (357h16m29.495480413s) [noise=true , address=77.90.31.18:4805 , piece_id=ZOPHHHDPANVCR24DYEVFTPTEI7QZWJC2CSKHTA3AIOE4LVSAIO7A, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.944393923 (357h16m29.495864385s) [noise=true , address=145.79.227.15:4874 , piece_id=HX222X63BP3XZJDWRUA66L25RB5OQC6QVO64KFE2YIVAASMH5V5Q, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.944915867 (357h16m29.496386369s) [noise=true , address=145.79.227.27:4690 , piece_id=QY2UEG6CDJQC7PK5Y4CV7NNWAO6OIU7G7AM4LOULWAYIIXP72DYQ, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.945511261 (357h16m29.496981712s) [noise=true , address=145.79.227.13:4804 , piece_id=MY352JGM6TNDPQRCT5NAQ5F43PQZFLCF3LAPBOXKERH4LKVXJMDA, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.946892241 (357h16m29.498362702s) [noise=true , address=145.79.227.31:4799 , piece_id=ORXIPX6OLF6KRZMILCLQLI6DJ4R3WWD2BG7COIFXCC3LTIJAR7CQ, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.948306093 (357h16m29.499776554s) [noise=true , address=145.79.227.9:4365 , piece_id=IVA4Z3LIMJDEZJAOQOMXHHK6YER6IJLIT6T5TYVALG4YKSOIITFA, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.949923172 (357h16m29.501393633s) [noise=true , address=77.90.31.10:4518 , piece_id=QGJDEHZDAG6WIVH2OI27WXBUTPGWJLYW7YUF626EHDXQUDK2GBCQ, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.951243466 (357h16m29.502713928s) [noise=true , address=45.91.116.9:4263 , piece_id=NWOUVIIGEJTZMBFVMGAHSAW2QXOVWQ3R6SVAFWKGZBLPI2CAJE4Q, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.952429815 (357h16m29.503900266s) [noise=true , address=45.91.116.8:4045 , piece_id=WY4ATDPUHVRHYJRQN3LMXJIKCFZIQRGD6DZL5LGZSHFDD2IL565A, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.953561660 (357h16m29.505032121s) [noise=true , address=145.79.227.20:4438 , piece_id=6YUL34NHDGETIJVH4IDTJ7OATWQDHDSS7STNYMIWDXIYDSKMVUHQ, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.953950061 (357h16m29.505420522s) [noise=true , address=145.79.227.18:4006 , piece_id=B3XG2EOIQVHMIDVCUSRTJPLWQFD7LMUKWCCNWKVO3VNOXYCSGUWQ, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:32.649721339 (357h16m31.2011918s) [upload=314482]: Next returned nil segment Nov 24 05:20:32.649739322 (357h16m31.201209804s) [upload=314482]: Waiting for error group managing segments... Nov 24 05:20:32.999561294 (357h16m31.551031755s) [noise=true , address=45.91.116.8:4045 , piece_id=WY4ATDPUHVRHYJRQN3LMXJIKCFZIQRGD6DZL5LGZSHFDD2IL565A, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:33.052276352 (357h16m31.603746803s) [noise=true , address=45.91.116.11:4118 , piece_id=SKTEMZIMVA3B2HC2Q3ZJI2JZXDJBI4TQ6HWCQU5SRX7B4Y2URGBQ, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:33.060891888 (357h16m31.612362339s) [noise=true , address=77.90.31.18:4805 , piece_id=ZOPHHHDPANVCR24DYEVFTPTEI7QZWJC2CSKHTA3AIOE4LVSAIO7A, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:33.096526542 (357h16m31.647997003s) [noise=true , address=45.91.116.9:4263 , piece_id=NWOUVIIGEJTZMBFVMGAHSAW2QXOVWQ3R6SVAFWKGZBLPI2CAJE4Q, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:33.108779213 (357h16m31.660249674s) [noise=true , address=77.90.31.10:4518 , piece_id=QGJDEHZDAG6WIVH2OI27WXBUTPGWJLYW7YUF626EHDXQUDK2GBCQ, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:33.127759234 (357h16m31.679229695s) [noise=true , address=77.90.31.12:4592 , piece_id=FGZSDMDUHHWA25KVCJ67FC75SAO7TTFOOKTXLMJEVHK3G25UOEJQ, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:33.149265876 (357h16m31.700736338s) [noise=true , address=145.79.227.27:4690 , piece_id=QY2UEG6CDJQC7PK5Y4CV7NNWAO6OIU7G7AM4LOULWAYIIXP72DYQ, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:34.189353961 (357h16m32.740824422s) [noise=true , address=145.79.227.24:4581 , piece_id=WGL7V2A7QYIB7W5PIQW6VGFVEXWZBVG6BZJEWRZFCR32NRPAF42A, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:34.317358835 (357h16m32.868829296s) [noise=true , address=145.79.227.9:4365 , piece_id=IVA4Z3LIMJDEZJAOQOMXHHK6YER6IJLIT6T5TYVALG4YKSOIITFA, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:34.801613418 (357h16m33.353083879s) [noise=true , address=145.79.227.20:4438 , piece_id=6YUL34NHDGETIJVH4IDTJ7OATWQDHDSS7STNYMIWDXIYDSKMVUHQ, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:35.533009581 (357h16m34.084480042s) [noise=true , address=145.79.227.31:4799 , piece_id=ORXIPX6OLF6KRZMILCLQLI6DJ4R3WWD2BG7COIFXCC3LTIJAR7CQ, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:35.881405926 (357h16m34.432876387s) [noise=true , address=145.79.227.18:4006 , piece_id=B3XG2EOIQVHMIDVCUSRTJPLWQFD7LMUKWCCNWKVO3VNOXYCSGUWQ, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:36.009138310 (357h16m34.560608772s) [noise=true , address=145.79.227.14:4839 , piece_id=4GCQV2V53YRT542J2H4BONZGE5LC7BV6GJ5NSLEDCNKAR3ZOIMCQ, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:36.009166294 (357h16m34.560636755s) [seg_pos={0 0}, upload=314482]: Segment reached optimal threshold of 13 pieces. Nov 24 05:20:36.009604859 (357h16m34.561075321s) [noise=true , address=145.79.227.15:4874 , piece_id=HX222X63BP3XZJDWRUA66L25RB5OQC6QVO64KFE2YIVAASMH5V5Q, seg_pos={0 0}, upload=314482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: read tcp 188.166.225.170:48954->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 24 05:20:36.009615309 (357h16m34.56108577s) [noise=true , address=145.79.227.15:4874 , piece_id=HX222X63BP3XZJDWRUA66L25RB5OQC6QVO64KFE2YIVAASMH5V5Q, seg_pos={0 0}, upload=314482]: Long tail cancellation due to optimal threshold Nov 24 05:20:36.009966337 (357h16m34.561436798s) [noise=true , address=145.79.227.13:4804 , piece_id=MY352JGM6TNDPQRCT5NAQ5F43PQZFLCF3LAPBOXKERH4LKVXJMDA, seg_pos={0 0}, upload=314482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: write tcp 188.166.225.170:49618->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:49618->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49618->145.79.227.13:4804: use of closed network connection Nov 24 05:20:36.010005041 (357h16m34.561475502s) [noise=true , address=145.79.227.13:4804 , piece_id=MY352JGM6TNDPQRCT5NAQ5F43PQZFLCF3LAPBOXKERH4LKVXJMDA, seg_pos={0 0}, upload=314482]: Long tail cancellation due to optimal threshold Nov 24 05:20:36.010055868 (357h16m34.561526339s) [noise=true , address=145.79.227.4:4186 , piece_id=WA6P7W3GAYWQQCN4WTLF2TPZEF4YH333KS3CRAYCSUDFXSETUYJA, seg_pos={0 0}, upload=314482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: write tcp 188.166.225.170:50504->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:50504->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50504->145.79.227.4:4186: use of closed network connection Nov 24 05:20:36.010069965 (357h16m34.561540426s) [noise=true , address=145.79.227.4:4186 , piece_id=WA6P7W3GAYWQQCN4WTLF2TPZEF4YH333KS3CRAYCSUDFXSETUYJA, seg_pos={0 0}, upload=314482]: Long tail cancellation due to optimal threshold Nov 24 05:20:36.010102216 (357h16m34.561572677s) [seg_pos={0 0}, upload=314482]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:20:36.010201435 (357h16m34.561671896s) [upload=314482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:20:36.010209561 (357h16m34.561680022s) [upload=314482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:20:36.488610929 (357h16m35.04008139s) [upload=314482]: Done uploading segments... Nov 24 05:20:36.488637500 (357h16m35.040107951s) [upload=314482]: Upload finished. err: <nil> Duration: 5.870504473s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.004474239s trace: https://jaeger.dev.storj.io/trace/68b6311f086e60c6 FAIL (2025-11-24 05:21:09 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.004506241s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.005014844s trace: https://jaeger.dev.storj.io/trace/747a47aa26cdc65f FAIL (2025-11-24 05:21:02 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.005122227s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.008377783s trace: https://jaeger.dev.storj.io/trace/68b6311f086e60c6 FAIL (2025-11-24 05:20:47 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.008401629s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................................X...X.........XX] duration: 30.002498072s trace: https://jaeger.dev.storj.io/trace/656c5eee210484ef FAIL (2025-11-24 05:20:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4 05:20:27.101694298 (357h16m25.653164749s) [upload=314482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:20:27.101700471 (357h16m25.653170922s) [upload=314482]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:20:30.941084752 (357h16m29.492555354s) [seg_pos={0 0}, upload=314482]: Begin upload segment... Nov 24 05:20:30.941151700 (357h16m29.492622452s) [seg_pos={0 0}, upload=314482]: Done begin upload segment. Nov 24 05:20:30.941192067 (357h16m29.492662528s) [noise=true , address=77.90.31.12:4592 , piece_id=FGZSDMDUHHWA25KVCJ67FC75SAO7TTFOOKTXLMJEVHK3G25UOEJQ, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.942182242 (357h16m29.493652703s) [noise=true , address=145.79.227.24:4581 , piece_id=WGL7V2A7QYIB7W5PIQW6VGFVEXWZBVG6BZJEWRZFCR32NRPAF42A, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.942590840 (357h16m29.494061301s) [noise=true , address=45.91.116.11:4118 , piece_id=SKTEMZIMVA3B2HC2Q3ZJI2JZXDJBI4TQ6HWCQU5SRX7B4Y2URGBQ, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.943065604 (357h16m29.494536065s) [noise=true , address=145.79.227.4:4186 , piece_id=WA6P7W3GAYWQQCN4WTLF2TPZEF4YH333KS3CRAYCSUDFXSETUYJA, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.943513267 (357h16m29.494983718s) [noise=true , address=145.79.227.14:4839 , piece_id=4GCQV2V53YRT542J2H4BONZGE5LC7BV6GJ5NSLEDCNKAR3ZOIMCQ, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.944009952 (357h16m29.495480413s) [noise=true , address=77.90.31.18:4805 , piece_id=ZOPHHHDPANVCR24DYEVFTPTEI7QZWJC2CSKHTA3AIOE4LVSAIO7A, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.944393923 (357h16m29.495864385s) [noise=true , address=145.79.227.15:4874 , piece_id=HX222X63BP3XZJDWRUA66L25RB5OQC6QVO64KFE2YIVAASMH5V5Q, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.944915867 (357h16m29.496386369s) [noise=true , address=145.79.227.27:4690 , piece_id=QY2UEG6CDJQC7PK5Y4CV7NNWAO6OIU7G7AM4LOULWAYIIXP72DYQ, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.945511261 (357h16m29.496981712s) [noise=true , address=145.79.227.13:4804 , piece_id=MY352JGM6TNDPQRCT5NAQ5F43PQZFLCF3LAPBOXKERH4LKVXJMDA, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.946892241 (357h16m29.498362702s) [noise=true , address=145.79.227.31:4799 , piece_id=ORXIPX6OLF6KRZMILCLQLI6DJ4R3WWD2BG7COIFXCC3LTIJAR7CQ, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.948306093 (357h16m29.499776554s) [noise=true , address=145.79.227.9:4365 , piece_id=IVA4Z3LIMJDEZJAOQOMXHHK6YER6IJLIT6T5TYVALG4YKSOIITFA, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.949923172 (357h16m29.501393633s) [noise=true , address=77.90.31.10:4518 , piece_id=QGJDEHZDAG6WIVH2OI27WXBUTPGWJLYW7YUF626EHDXQUDK2GBCQ, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.951243466 (357h16m29.502713928s) [noise=true , address=45.91.116.9:4263 , piece_id=NWOUVIIGEJTZMBFVMGAHSAW2QXOVWQ3R6SVAFWKGZBLPI2CAJE4Q, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.952429815 (357h16m29.503900266s) [noise=true , address=45.91.116.8:4045 , piece_id=WY4ATDPUHVRHYJRQN3LMXJIKCFZIQRGD6DZL5LGZSHFDD2IL565A, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.953561660 (357h16m29.505032121s) [noise=true , address=145.79.227.20:4438 , piece_id=6YUL34NHDGETIJVH4IDTJ7OATWQDHDSS7STNYMIWDXIYDSKMVUHQ, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:30.953950061 (357h16m29.505420522s) [noise=true , address=145.79.227.18:4006 , piece_id=B3XG2EOIQVHMIDVCUSRTJPLWQFD7LMUKWCCNWKVO3VNOXYCSGUWQ, seg_pos={0 0}, upload=314482]: Uploading piece... Nov 24 05:20:32.649721339 (357h16m31.2011918s) [upload=314482]: Next returned nil segment Nov 24 05:20:32.649739322 (357h16m31.201209804s) [upload=314482]: Waiting for error group managing segments... Nov 24 05:20:32.999561294 (357h16m31.551031755s) [noise=true , address=45.91.116.8:4045 , piece_id=WY4ATDPUHVRHYJRQN3LMXJIKCFZIQRGD6DZL5LGZSHFDD2IL565A, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:33.052276352 (357h16m31.603746803s) [noise=true , address=45.91.116.11:4118 , piece_id=SKTEMZIMVA3B2HC2Q3ZJI2JZXDJBI4TQ6HWCQU5SRX7B4Y2URGBQ, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:33.060891888 (357h16m31.612362339s) [noise=true , address=77.90.31.18:4805 , piece_id=ZOPHHHDPANVCR24DYEVFTPTEI7QZWJC2CSKHTA3AIOE4LVSAIO7A, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:33.096526542 (357h16m31.647997003s) [noise=true , address=45.91.116.9:4263 , piece_id=NWOUVIIGEJTZMBFVMGAHSAW2QXOVWQ3R6SVAFWKGZBLPI2CAJE4Q, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:33.108779213 (357h16m31.660249674s) [noise=true , address=77.90.31.10:4518 , piece_id=QGJDEHZDAG6WIVH2OI27WXBUTPGWJLYW7YUF626EHDXQUDK2GBCQ, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:33.127759234 (357h16m31.679229695s) [noise=true , address=77.90.31.12:4592 , piece_id=FGZSDMDUHHWA25KVCJ67FC75SAO7TTFOOKTXLMJEVHK3G25UOEJQ, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:33.149265876 (357h16m31.700736338s) [noise=true , address=145.79.227.27:4690 , piece_id=QY2UEG6CDJQC7PK5Y4CV7NNWAO6OIU7G7AM4LOULWAYIIXP72DYQ, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:34.189353961 (357h16m32.740824422s) [noise=true , address=145.79.227.24:4581 , piece_id=WGL7V2A7QYIB7W5PIQW6VGFVEXWZBVG6BZJEWRZFCR32NRPAF42A, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:34.317358835 (357h16m32.868829296s) [noise=true , address=145.79.227.9:4365 , piece_id=IVA4Z3LIMJDEZJAOQOMXHHK6YER6IJLIT6T5TYVALG4YKSOIITFA, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:34.801613418 (357h16m33.353083879s) [noise=true , address=145.79.227.20:4438 , piece_id=6YUL34NHDGETIJVH4IDTJ7OATWQDHDSS7STNYMIWDXIYDSKMVUHQ, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:35.533009581 (357h16m34.084480042s) [noise=true , address=145.79.227.31:4799 , piece_id=ORXIPX6OLF6KRZMILCLQLI6DJ4R3WWD2BG7COIFXCC3LTIJAR7CQ, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:35.881405926 (357h16m34.432876387s) [noise=true , address=145.79.227.18:4006 , piece_id=B3XG2EOIQVHMIDVCUSRTJPLWQFD7LMUKWCCNWKVO3VNOXYCSGUWQ, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:36.009138310 (357h16m34.560608772s) [noise=true , address=145.79.227.14:4839 , piece_id=4GCQV2V53YRT542J2H4BONZGE5LC7BV6GJ5NSLEDCNKAR3ZOIMCQ, seg_pos={0 0}, upload=314482]: Done uploading piece. err: <nil> Nov 24 05:20:36.009166294 (357h16m34.560636755s) [seg_pos={0 0}, upload=314482]: Segment reached optimal threshold of 13 pieces. Nov 24 05:20:36.009604859 (357h16m34.561075321s) [noise=true , address=145.79.227.15:4874 , piece_id=HX222X63BP3XZJDWRUA66L25RB5OQC6QVO64KFE2YIVAASMH5V5Q, seg_pos={0 0}, upload=314482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: read tcp 188.166.225.170:48954->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 24 05:20:36.009615309 (357h16m34.56108577s) [noise=true , address=145.79.227.15:4874 , piece_id=HX222X63BP3XZJDWRUA66L25RB5OQC6QVO64KFE2YIVAASMH5V5Q, seg_pos={0 0}, upload=314482]: Long tail cancellation due to optimal threshold Nov 24 05:20:36.009966337 (357h16m34.561436798s) [noise=true , address=145.79.227.13:4804 , piece_id=MY352JGM6TNDPQRCT5NAQ5F43PQZFLCF3LAPBOXKERH4LKVXJMDA, seg_pos={0 0}, upload=314482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: write tcp 188.166.225.170:49618->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:49618->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49618->145.79.227.13:4804: use of closed network connection Nov 24 05:20:36.010005041 (357h16m34.561475502s) [noise=true , address=145.79.227.13:4804 , piece_id=MY352JGM6TNDPQRCT5NAQ5F43PQZFLCF3LAPBOXKERH4LKVXJMDA, seg_pos={0 0}, upload=314482]: Long tail cancellation due to optimal threshold Nov 24 05:20:36.010055868 (357h16m34.561526339s) [noise=true , address=145.79.227.4:4186 , piece_id=WA6P7W3GAYWQQCN4WTLF2TPZEF4YH333KS3CRAYCSUDFXSETUYJA, seg_pos={0 0}, upload=314482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: write tcp 188.166.225.170:50504->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:50504->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50504->145.79.227.4:4186: use of closed network connection Nov 24 05:20:36.010069965 (357h16m34.561540426s) [noise=true , address=145.79.227.4:4186 , piece_id=WA6P7W3GAYWQQCN4WTLF2TPZEF4YH333KS3CRAYCSUDFXSETUYJA, seg_pos={0 0}, upload=314482]: Long tail cancellation due to optimal threshold Nov 24 05:20:36.010102216 (357h16m34.561572677s) [seg_pos={0 0}, upload=314482]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:20:36.010201435 (357h16m34.561671896s) [upload=314482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:20:36.010209561 (357h16m34.561680022s) [upload=314482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:20:36.488610929 (357h16m35.04008139s) [upload=314482]: Done uploading segments... Nov 24 05:20:36.488637500 (357h16m35.040107951s) [upload=314482]: Upload finished. err: <nil> Duration: 30.002519913s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001484582s trace: https://jaeger.dev.storj.io/trace/747a47aa26cdc65f FAIL (2025-11-24 05:20:29 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001517825s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002290817s trace: https://jaeger.dev.storj.io/trace/68b6311f086e60c6 FAIL (2025-11-24 05:20:25 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 30.002315986s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................................X...X.........X] duration: 30.002929075s trace: https://jaeger.dev.storj.io/trace/797fdc3870ab1dfa FAIL (2025-11-24 05:20:07 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== 74346075s) [upload=314475]: Uploading segments... Nov 24 05:19:50.122924176 (357h15m48.674394627s) [upload=314475]: Got next segment. Inline: false Nov 24 05:19:50.122957709 (357h15m48.674428171s) [upload=314475]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:19:50.122964011 (357h15m48.674434462s) [upload=314475]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:19:53.051203880 (357h15m51.602674351s) [seg_pos={0 0}, upload=314475]: Begin upload segment... Nov 24 05:19:53.051266128 (357h15m51.602736589s) [seg_pos={0 0}, upload=314475]: Done begin upload segment. Nov 24 05:19:53.051312366 (357h15m51.602782827s) [noise=true , address=145.79.227.26:4654 , piece_id=XLN4X3SDHCCRX46HGAI6U6TRPVZEHV5DDLBVXPJOBIYWZLGA4RNQ, seg_pos={0 0}, upload=314475]: Uploading piece... Nov 24 05:19:53.051336733 (357h15m51.602807194s) [noise=true , address=145.79.227.14:4838 , piece_id=JCVRRPR4TJKFGFDAYCOITIBB7MLERJOZPEFA4KINS4I7X34JJ3SA, seg_pos={0 0}, upload=314475]: Uploading piece... Nov 24 05:19:53.051395414 (357h15m51.602865875s) [noise=true , address=145.79.227.21:4475 , piece_id=B45QCCXRU3R3ORT5MIRANH33PZZKO5CBMEJI6ISLMBFOL5Y66PNA, seg_pos={0 0}, upload=314475]: Uploading piece... Nov 24 05:19:53.052209845 (357h15m51.603680306s) [noise=true , address=145.79.227.31:4797 , piece_id=PUPORR4OGST7IOKJTBS6SLDJHJB5F732CDLLMLLBKOKVBAAXS4JA, seg_pos={0 0}, upload=314475]: Uploading piece... Nov 24 05:19:53.052365030 (357h15m51.603835491s) [noise=true , address=145.79.227.10:4401 , piece_id=HYFNYHI3M3FXPN55Y6LPASDSIN7SXTELHJMPK6RNTBLYIDYIJE7A, seg_pos={0 0}, upload=314475]: Uploading piece... Nov 24 05:19:53.052685170 (357h15m51.604155631s) [noise=true , address=145.79.227.8:4331 , piece_id=Q7GPSWSCFE6AVZH6TQEQWRIY7Z5VRPVE7E4DGR6EUZLHKZVQUXYQ, seg_pos={0 0}, upload=314475]: Uploading piece... Nov 24 05:19:53.052973429 (357h15m51.60444388s) [noise=true , address=145.79.227.23:4546 , piece_id=JFGSH7XLDNI7L55O7ZD34KNDJOXRWQ7L4YPNCC7KADGVE7O6ZNFQ, seg_pos={0 0}, upload=314475]: Uploading piece... Nov 24 05:19:53.053250517 (357h15m51.604720978s) [noise=true , address=77.90.31.10:4519 , piece_id=6Q6VPX2DGR4D2RTLWYTYNJ37WDU42ILUFXQ5MX3QSMLIBRTOADQA, seg_pos={0 0}, upload=314475]: Uploading piece... Nov 24 05:19:53.053593088 (357h15m51.60506354s) [noise=true , address=145.79.227.13:4801 , piece_id=LZR3WGJDPFELLPCFT7B74W64HK3HHW3N2VYUUVLPR5UQWF3EX6BQ, seg_pos={0 0}, upload=314475]: Uploading piece... Nov 24 05:19:53.053995796 (357h15m51.605466257s) [noise=true , address=145.79.227.25:4619 , piece_id=UULQO2VHHMIK4DUTTXOQ46DRX4HQUJGHZSMSBN5WTJYMJXZAT6YA, seg_pos={0 0}, upload=314475]: Uploading piece... Nov 24 05:19:53.054182060 (357h15m51.605652521s) [noise=true , address=145.79.227.27:4689 , piece_id=WYII7Z6HZGPBOIX3T7HNZK4GAPCW6NQAKMI4BHOEPR2SKEC2I3IA, seg_pos={0 0}, upload=314475]: Uploading piece... Nov 24 05:19:53.054408091 (357h15m51.605878552s) [noise=true , address=145.79.227.9:4367 , piece_id=P2TH4RBJLXYWQY4Y4GY7KJ5FLVN4L26XZC4223OEDQ6C5MN2LFHA, seg_pos={0 0}, upload=314475]: Uploading piece... Nov 24 05:19:53.054583074 (357h15m51.606053545s) [noise=true , address=145.79.227.6:4257 , piece_id=3AZGGQ5AHLLA6GNGSLQ5S3BRP2L5MGVLO6XKC32Y7LCZUFL7TRLQ, seg_pos={0 0}, upload=314475]: Uploading piece... Nov 24 05:19:53.054972215 (357h15m51.606442676s) [noise=true , address=145.79.227.28:4727 , piece_id=IL4I3NRY5SXZ5CNISTBCGZIQUWEYMLHAKA5UWPPPHW56MR5DWEBA, seg_pos={0 0}, upload=314475]: Uploading piece... Nov 24 05:19:53.056257422 (357h15m51.607727883s) [noise=true , address=45.91.116.12:4227 , piece_id=G3Y6JR3EM6GU6UKKUGNI4SMDLDQFJMNYRBKTQMJKT4GXMWDSEXTA, seg_pos={0 0}, upload=314475]: Uploading piece... Nov 24 05:19:53.057051134 (357h15m51.608521595s) [noise=true , address=145.79.227.16:4910 , piece_id=LTX56OIAAXHZCZYRQ557HGAJED3KOCIG4NLMPYOHU7VCKFOFERLQ, seg_pos={0 0}, upload=314475]: Uploading piece... Nov 24 05:19:54.896673418 (357h15m53.448143879s) [upload=314475]: Next returned nil segment Nov 24 05:19:54.896691843 (357h15m53.448162305s) [upload=314475]: Waiting for error group managing segments... Nov 24 05:19:55.186993037 (357h15m53.738463488s) [noise=true , address=77.90.31.10:4519 , piece_id=6Q6VPX2DGR4D2RTLWYTYNJ37WDU42ILUFXQ5MX3QSMLIBRTOADQA, seg_pos={0 0}, upload=314475]: Done uploading piece. err: <nil> Nov 24 05:19:55.210899969 (357h15m53.76237042s) [noise=true , address=45.91.116.12:4227 , piece_id=G3Y6JR3EM6GU6UKKUGNI4SMDLDQFJMNYRBKTQMJKT4GXMWDSEXTA, seg_pos={0 0}, upload=314475]: Done uploading piece. err: <nil> Nov 24 05:19:55.264071297 (357h15m53.815541758s) [noise=true , address=145.79.227.9:4367 , piece_id=P2TH4RBJLXYWQY4Y4GY7KJ5FLVN4L26XZC4223OEDQ6C5MN2LFHA, seg_pos={0 0}, upload=314475]: Done uploading piece. err: <nil> Nov 24 05:19:55.326868503 (357h15m53.878338965s) [noise=true , address=145.79.227.14:4838 , piece_id=JCVRRPR4TJKFGFDAYCOITIBB7MLERJOZPEFA4KINS4I7X34JJ3SA, seg_pos={0 0}, upload=314475]: Done uploading piece. err: <nil> Nov 24 05:19:55.386853324 (357h15m53.938323775s) [noise=true , address=145.79.227.21:4475 , piece_id=B45QCCXRU3R3ORT5MIRANH33PZZKO5CBMEJI6ISLMBFOL5Y66PNA, seg_pos={0 0}, upload=314475]: Done uploading piece. err: <nil> Nov 24 05:19:55.391414489 (357h15m53.94288495s) [noise=true , address=145.79.227.25:4619 , piece_id=UULQO2VHHMIK4DUTTXOQ46DRX4HQUJGHZSMSBN5WTJYMJXZAT6YA, seg_pos={0 0}, upload=314475]: Done uploading piece. err: <nil> Nov 24 05:19:55.427978754 (357h15m53.979449205s) [noise=true , address=145.79.227.23:4546 , piece_id=JFGSH7XLDNI7L55O7ZD34KNDJOXRWQ7L4YPNCC7KADGVE7O6ZNFQ, seg_pos={0 0}, upload=314475]: Done uploading piece. err: <nil> Nov 24 05:19:56.005878774 (357h15m54.557349236s) [noise=true , address=145.79.227.28:4727 , piece_id=IL4I3NRY5SXZ5CNISTBCGZIQUWEYMLHAKA5UWPPPHW56MR5DWEBA, seg_pos={0 0}, upload=314475]: Done uploading piece. err: <nil> Nov 24 05:19:56.042002630 (357h15m54.593473091s) [noise=true , address=145.79.227.6:4257 , piece_id=3AZGGQ5AHLLA6GNGSLQ5S3BRP2L5MGVLO6XKC32Y7LCZUFL7TRLQ, seg_pos={0 0}, upload=314475]: Done uploading piece. err: <nil> Nov 24 05:19:57.346167813 (357h15m55.897638274s) [noise=true , address=145.79.227.10:4401 , piece_id=HYFNYHI3M3FXPN55Y6LPASDSIN7SXTELHJMPK6RNTBLYIDYIJE7A, seg_pos={0 0}, upload=314475]: Done uploading piece. err: <nil> Nov 24 05:19:58.177626793 (357h15m56.729097254s) [noise=true , address=145.79.227.31:4797 , piece_id=PUPORR4OGST7IOKJTBS6SLDJHJB5F732CDLLMLLBKOKVBAAXS4JA, seg_pos={0 0}, upload=314475]: Done uploading piece. err: <nil> Nov 24 05:19:59.874616571 (357h15m58.426087032s) [noise=true , address=145.79.227.13:4801 , piece_id=LZR3WGJDPFELLPCFT7B74W64HK3HHW3N2VYUUVLPR5UQWF3EX6BQ, seg_pos={0 0}, upload=314475]: Done uploading piece. err: <nil> Nov 24 05:20:02.248841430 (357h16m0.800311901s) [noise=true , address=145.79.227.8:4331 , piece_id=Q7GPSWSCFE6AVZH6TQEQWRIY7Z5VRPVE7E4DGR6EUZLHKZVQUXYQ, seg_pos={0 0}, upload=314475]: Done uploading piece. err: <nil> Nov 24 05:20:02.248869564 (357h16m0.800340025s) [seg_pos={0 0}, upload=314475]: Segment reached optimal threshold of 13 pieces. Nov 24 05:20:02.249112086 (357h16m0.800582547s) [noise=true , address=145.79.227.27:4689 , piece_id=WYII7Z6HZGPBOIX3T7HNZK4GAPCW6NQAKMI4BHOEPR2SKEC2I3IA, seg_pos={0 0}, upload=314475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ): protocol: write tcp 188.166.225.170:48650->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:48650->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48650->145.79.227.27:4689: use of closed network connection Nov 24 05:20:02.249120181 (357h16m0.800590642s) [noise=true , address=145.79.227.27:4689 , piece_id=WYII7Z6HZGPBOIX3T7HNZK4GAPCW6NQAKMI4BHOEPR2SKEC2I3IA, seg_pos={0 0}, upload=314475]: Long tail cancellation due to optimal threshold Nov 24 05:20:02.249407488 (357h16m0.800877949s) [noise=true , address=145.79.227.16:4910 , piece_id=LTX56OIAAXHZCZYRQ557HGAJED3KOCIG4NLMPYOHU7VCKFOFERLQ, seg_pos={0 0}, upload=314475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:56574->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 24 05:20:02.249415703 (357h16m0.800886174s) [noise=true , address=145.79.227.16:4910 , piece_id=LTX56OIAAXHZCZYRQ557HGAJED3KOCIG4NLMPYOHU7VCKFOFERLQ, seg_pos={0 0}, upload=314475]: Long tail cancellation due to optimal threshold Nov 24 05:20:02.249745021 (357h16m0.801215482s) [noise=true , address=145.79.227.26:4654 , piece_id=XLN4X3SDHCCRX46HGAI6U6TRPVZEHV5DDLBVXPJOBIYWZLGA4RNQ, seg_pos={0 0}, upload=314475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: read tcp 188.166.225.170:52370->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Nov 24 05:20:02.249842416 (357h16m0.801312877s) [noise=true , address=145.79.227.26:4654 , piece_id=XLN4X3SDHCCRX46HGAI6U6TRPVZEHV5DDLBVXPJOBIYWZLGA4RNQ, seg_pos={0 0}, upload=314475]: Long tail cancellation due to optimal threshold Nov 24 05:20:02.249867032 (357h16m0.801337494s) [seg_pos={0 0}, upload=314475]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:20:02.250001970 (357h16m0.801472431s) [upload=314475]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:20:02.250009804 (357h16m0.801480265s) [upload=314475]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:20:07.463504759 (357h16m6.01497524s) [upload=314475]: Done uploading segments... Nov 24 05:20:07.463533363 (357h16m6.015003825s) [upload=314475]: Upload finished. err: <nil> Duration: 30.002957789s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X...X.........] duration: 5.883228251s trace: https://jaeger.dev.storj.io/trace/514639791a8f826b FAIL (2025-11-24 05:19:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ) context deadline exceeded; piecestore: (Node ID: 12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, Piece ID: TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: 3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA) context deadline exceeded ======LOGS====== Nov 24 05:12:52.682445011 (357h8m51.233915462s) [upload=314373]: Starting upload Nov 24 05:12:52.682548268 (357h8m51.234018729s) [upload=314373]: Done starting upload Nov 24 05:12:52.682618251 (357h8m51.234088712s) [upload=314373]: Uploading segments... Nov 24 05:12:52.682764169 (357h8m51.23423466s) [upload=314373]: Got next segment. Inline: false Nov 24 05:12:52.682809646 (357h8m51.234280107s) [upload=314373]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:12:52.682816379 (357h8m51.23428684s) [upload=314373]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:12:53.903345875 (357h8m52.454816346s) [seg_pos={0 0}, upload=314373]: Begin upload segment... Nov 24 05:12:53.903444022 (357h8m52.454914523s) [seg_pos={0 0}, upload=314373]: Done begin upload segment. Nov 24 05:12:53.903485471 (357h8m52.454955932s) [noise=true , address=45.91.116.6:4973 , piece_id=YFXRU76I3BSIAI3DKFRFERTVMJLR2CENQHYP3TMUCDBKBLWGGRYA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.903511801 (357h8m52.454982262s) [noise=true , address=77.90.31.14:4661 , piece_id=KLSYXSHBESAOQKDM7UIEJRBEU47LAZ4QLXQ7IIRHMGRHQDOTTGSQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.904736563 (357h8m52.456207024s) [noise=true , address=145.79.227.23:4547 , piece_id=77DA6ZG5K7FIM5VVFGLLIU3ZFPFSE6LYXA5ICFT32TGPTE4E6EWA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.905132177 (357h8m52.456602628s) [noise=true , address=145.79.227.6:4257 , piece_id=4YPKUHVNKCB25N5VNGRUAHIEAWOEIZL6FIM66XZGAI4SGNT6YV5Q, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.905529643 (357h8m52.457000104s) [noise=true , address=145.79.227.25:4619 , piece_id=Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906023413 (357h8m52.457493874s) [noise=true , address=145.79.227.22:4511 , piece_id=TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906454855 (357h8m52.457925306s) [noise=true , address=145.79.227.21:4474 , piece_id=3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906891877 (357h8m52.458362338s) [noise=true , address=145.79.227.10:4402 , piece_id=JT3O5M6VTUIFKBZKLTCBHTQWCOEYNWF4G2VW6DR73MVPIEAERLBQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.908246807 (357h8m52.459717268s) [noise=true , address=145.79.227.15:4875 , piece_id=VUFSGD36YPWJ32JILLC3ATEW2NYLIA4H3ROPTEFYXQQNXYPUC6VA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.908390811 (357h8m52.459861272s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.909633959 (357h8m52.46110442s) [noise=true , address=145.79.227.24:4582 , piece_id=EXSXMMBRJ4AGORCP4YYSET432YNOTBYOU32O42BZOO42LYOGBJJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.910221468 (357h8m52.461691929s) [noise=true , address=77.90.31.13:4628 , piece_id=TO7ZMNG64OQLA4AFEZKTSBPPB7DGLE72Q2DHZRWMGN54X2FQ47TA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.911006863 (357h8m52.462477315s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.911862011 (357h8m52.463332462s) [noise=true , address=45.91.116.4:4937 , piece_id=SL3XPUODGLVL5SNMZU7A6BT57IQ7GVGZE343ZJVNSFWC5TX3GVVQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.912556263 (357h8m52.464026714s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.913294699 (357h8m52.46476516s) [noise=true , address=45.91.116.11:4118 , piece_id=NEVUNIGHGP5AQPONBJ5J3IJCVJTUYGKHQNITKVG2ZABNTUWWZIBA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:55.574621520 (357h8m54.126091981s) [upload=314373]: Next returned nil segment Nov 24 05:12:55.574639704 (357h8m54.126110165s) [upload=314373]: Waiting for error group managing segments... Nov 24 05:12:55.966933719 (357h8m54.51840418s) [noise=true , address=77.90.31.14:4661 , piece_id=KLSYXSHBESAOQKDM7UIEJRBEU47LAZ4QLXQ7IIRHMGRHQDOTTGSQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:55.981186689 (357h8m54.53265714s) [noise=true , address=45.91.116.11:4118 , piece_id=NEVUNIGHGP5AQPONBJ5J3IJCVJTUYGKHQNITKVG2ZABNTUWWZIBA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.045206764 (357h8m54.596677225s) [noise=true , address=45.91.116.6:4973 , piece_id=YFXRU76I3BSIAI3DKFRFERTVMJLR2CENQHYP3TMUCDBKBLWGGRYA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.047372809 (357h8m54.59884327s) [noise=true , address=77.90.31.13:4628 , piece_id=TO7ZMNG64OQLA4AFEZKTSBPPB7DGLE72Q2DHZRWMGN54X2FQ47TA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.056652799 (357h8m54.60812326s) [noise=true , address=145.79.227.6:4257 , piece_id=4YPKUHVNKCB25N5VNGRUAHIEAWOEIZL6FIM66XZGAI4SGNT6YV5Q, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.059483058 (357h8m54.610953519s) [noise=true , address=145.79.227.21:4474 , piece_id=3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.070599767 (357h8m54.622070228s) [noise=true , address=145.79.227.10:4402 , piece_id=JT3O5M6VTUIFKBZKLTCBHTQWCOEYNWF4G2VW6DR73MVPIEAERLBQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.074296216 (357h8m54.625766677s) [noise=true , address=45.91.116.4:4937 , piece_id=SL3XPUODGLVL5SNMZU7A6BT57IQ7GVGZE343ZJVNSFWC5TX3GVVQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.081602177 (357h8m54.633072639s) [noise=true , address=145.79.227.22:4511 , piece_id=TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.082195698 (357h8m54.633666149s) [noise=true , address=145.79.227.23:4547 , piece_id=77DA6ZG5K7FIM5VVFGLLIU3ZFPFSE6LYXA5ICFT32TGPTE4E6EWA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.089421227 (357h8m54.640891688s) [noise=true , address=145.79.227.24:4582 , piece_id=EXSXMMBRJ4AGORCP4YYSET432YNOTBYOU32O42BZOO42LYOGBJJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.094493995 (357h8m54.645964446s) [noise=true , address=145.79.227.15:4875 , piece_id=VUFSGD36YPWJ32JILLC3ATEW2NYLIA4H3ROPTEFYXQQNXYPUC6VA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.111919777 (357h8m54.663390248s) [noise=true , address=145.79.227.25:4619 , piece_id=Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.111947239 (357h8m54.6634177s) [seg_pos={0 0}, upload=314373]: Segment reached optimal threshold of 13 pieces. Nov 24 05:12:56.112360226 (357h8m54.663830707s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112371026 (357h8m54.663841487s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112778422 (357h8m54.664248883s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112789903 (357h8m54.664260364s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112805814 (357h8m54.664276265s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112813649 (357h8m54.66428411s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112892599 (357h8m54.66436305s) [seg_pos={0 0}, upload=314373]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:12:56.113070117 (357h8m54.664540588s) [upload=314373]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:12:56.113079936 (357h8m54.664550387s) [upload=314373]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:12:56.609050819 (357h8m55.16052127s) [upload=314373]: Done uploading segments... Nov 24 05:12:56.609072030 (357h8m55.160542491s) [upload=314373]: Upload finished. err: <nil> Duration: 5.883255734s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X...X........] duration: 5.833677812s trace: https://jaeger.dev.storj.io/trace/514639791a8f826b FAIL (2025-11-24 05:18:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ) context deadline exceeded; piecestore: (Node ID: 12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, Piece ID: TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: 3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA) context deadline exceeded ======LOGS====== Nov 24 05:12:52.682445011 (357h8m51.233915462s) [upload=314373]: Starting upload Nov 24 05:12:52.682548268 (357h8m51.234018729s) [upload=314373]: Done starting upload Nov 24 05:12:52.682618251 (357h8m51.234088712s) [upload=314373]: Uploading segments... Nov 24 05:12:52.682764169 (357h8m51.23423466s) [upload=314373]: Got next segment. Inline: false Nov 24 05:12:52.682809646 (357h8m51.234280107s) [upload=314373]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:12:52.682816379 (357h8m51.23428684s) [upload=314373]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:12:53.903345875 (357h8m52.454816346s) [seg_pos={0 0}, upload=314373]: Begin upload segment... Nov 24 05:12:53.903444022 (357h8m52.454914523s) [seg_pos={0 0}, upload=314373]: Done begin upload segment. Nov 24 05:12:53.903485471 (357h8m52.454955932s) [noise=true , address=45.91.116.6:4973 , piece_id=YFXRU76I3BSIAI3DKFRFERTVMJLR2CENQHYP3TMUCDBKBLWGGRYA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.903511801 (357h8m52.454982262s) [noise=true , address=77.90.31.14:4661 , piece_id=KLSYXSHBESAOQKDM7UIEJRBEU47LAZ4QLXQ7IIRHMGRHQDOTTGSQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.904736563 (357h8m52.456207024s) [noise=true , address=145.79.227.23:4547 , piece_id=77DA6ZG5K7FIM5VVFGLLIU3ZFPFSE6LYXA5ICFT32TGPTE4E6EWA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.905132177 (357h8m52.456602628s) [noise=true , address=145.79.227.6:4257 , piece_id=4YPKUHVNKCB25N5VNGRUAHIEAWOEIZL6FIM66XZGAI4SGNT6YV5Q, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.905529643 (357h8m52.457000104s) [noise=true , address=145.79.227.25:4619 , piece_id=Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906023413 (357h8m52.457493874s) [noise=true , address=145.79.227.22:4511 , piece_id=TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906454855 (357h8m52.457925306s) [noise=true , address=145.79.227.21:4474 , piece_id=3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906891877 (357h8m52.458362338s) [noise=true , address=145.79.227.10:4402 , piece_id=JT3O5M6VTUIFKBZKLTCBHTQWCOEYNWF4G2VW6DR73MVPIEAERLBQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.908246807 (357h8m52.459717268s) [noise=true , address=145.79.227.15:4875 , piece_id=VUFSGD36YPWJ32JILLC3ATEW2NYLIA4H3ROPTEFYXQQNXYPUC6VA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.908390811 (357h8m52.459861272s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.909633959 (357h8m52.46110442s) [noise=true , address=145.79.227.24:4582 , piece_id=EXSXMMBRJ4AGORCP4YYSET432YNOTBYOU32O42BZOO42LYOGBJJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.910221468 (357h8m52.461691929s) [noise=true , address=77.90.31.13:4628 , piece_id=TO7ZMNG64OQLA4AFEZKTSBPPB7DGLE72Q2DHZRWMGN54X2FQ47TA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.911006863 (357h8m52.462477315s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.911862011 (357h8m52.463332462s) [noise=true , address=45.91.116.4:4937 , piece_id=SL3XPUODGLVL5SNMZU7A6BT57IQ7GVGZE343ZJVNSFWC5TX3GVVQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.912556263 (357h8m52.464026714s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.913294699 (357h8m52.46476516s) [noise=true , address=45.91.116.11:4118 , piece_id=NEVUNIGHGP5AQPONBJ5J3IJCVJTUYGKHQNITKVG2ZABNTUWWZIBA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:55.574621520 (357h8m54.126091981s) [upload=314373]: Next returned nil segment Nov 24 05:12:55.574639704 (357h8m54.126110165s) [upload=314373]: Waiting for error group managing segments... Nov 24 05:12:55.966933719 (357h8m54.51840418s) [noise=true , address=77.90.31.14:4661 , piece_id=KLSYXSHBESAOQKDM7UIEJRBEU47LAZ4QLXQ7IIRHMGRHQDOTTGSQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:55.981186689 (357h8m54.53265714s) [noise=true , address=45.91.116.11:4118 , piece_id=NEVUNIGHGP5AQPONBJ5J3IJCVJTUYGKHQNITKVG2ZABNTUWWZIBA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.045206764 (357h8m54.596677225s) [noise=true , address=45.91.116.6:4973 , piece_id=YFXRU76I3BSIAI3DKFRFERTVMJLR2CENQHYP3TMUCDBKBLWGGRYA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.047372809 (357h8m54.59884327s) [noise=true , address=77.90.31.13:4628 , piece_id=TO7ZMNG64OQLA4AFEZKTSBPPB7DGLE72Q2DHZRWMGN54X2FQ47TA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.056652799 (357h8m54.60812326s) [noise=true , address=145.79.227.6:4257 , piece_id=4YPKUHVNKCB25N5VNGRUAHIEAWOEIZL6FIM66XZGAI4SGNT6YV5Q, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.059483058 (357h8m54.610953519s) [noise=true , address=145.79.227.21:4474 , piece_id=3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.070599767 (357h8m54.622070228s) [noise=true , address=145.79.227.10:4402 , piece_id=JT3O5M6VTUIFKBZKLTCBHTQWCOEYNWF4G2VW6DR73MVPIEAERLBQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.074296216 (357h8m54.625766677s) [noise=true , address=45.91.116.4:4937 , piece_id=SL3XPUODGLVL5SNMZU7A6BT57IQ7GVGZE343ZJVNSFWC5TX3GVVQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.081602177 (357h8m54.633072639s) [noise=true , address=145.79.227.22:4511 , piece_id=TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.082195698 (357h8m54.633666149s) [noise=true , address=145.79.227.23:4547 , piece_id=77DA6ZG5K7FIM5VVFGLLIU3ZFPFSE6LYXA5ICFT32TGPTE4E6EWA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.089421227 (357h8m54.640891688s) [noise=true , address=145.79.227.24:4582 , piece_id=EXSXMMBRJ4AGORCP4YYSET432YNOTBYOU32O42BZOO42LYOGBJJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.094493995 (357h8m54.645964446s) [noise=true , address=145.79.227.15:4875 , piece_id=VUFSGD36YPWJ32JILLC3ATEW2NYLIA4H3ROPTEFYXQQNXYPUC6VA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.111919777 (357h8m54.663390248s) [noise=true , address=145.79.227.25:4619 , piece_id=Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.111947239 (357h8m54.6634177s) [seg_pos={0 0}, upload=314373]: Segment reached optimal threshold of 13 pieces. Nov 24 05:12:56.112360226 (357h8m54.663830707s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112371026 (357h8m54.663841487s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112778422 (357h8m54.664248883s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112789903 (357h8m54.664260364s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112805814 (357h8m54.664276265s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112813649 (357h8m54.66428411s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112892599 (357h8m54.66436305s) [seg_pos={0 0}, upload=314373]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:12:56.113070117 (357h8m54.664540588s) [upload=314373]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:12:56.113079936 (357h8m54.664550387s) [upload=314373]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:12:56.609050819 (357h8m55.16052127s) [upload=314373]: Done uploading segments... Nov 24 05:12:56.609072030 (357h8m55.160542491s) [upload=314373]: Upload finished. err: <nil> Duration: 5.833705105s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X...X.......] duration: 5.80068297s trace: https://jaeger.dev.storj.io/trace/514639791a8f826b FAIL (2025-11-24 05:18:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ) context deadline exceeded; piecestore: (Node ID: 12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, Piece ID: TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: 3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA) context deadline exceeded ======LOGS====== Nov 24 05:12:52.682445011 (357h8m51.233915462s) [upload=314373]: Starting upload Nov 24 05:12:52.682548268 (357h8m51.234018729s) [upload=314373]: Done starting upload Nov 24 05:12:52.682618251 (357h8m51.234088712s) [upload=314373]: Uploading segments... Nov 24 05:12:52.682764169 (357h8m51.23423466s) [upload=314373]: Got next segment. Inline: false Nov 24 05:12:52.682809646 (357h8m51.234280107s) [upload=314373]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:12:52.682816379 (357h8m51.23428684s) [upload=314373]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:12:53.903345875 (357h8m52.454816346s) [seg_pos={0 0}, upload=314373]: Begin upload segment... Nov 24 05:12:53.903444022 (357h8m52.454914523s) [seg_pos={0 0}, upload=314373]: Done begin upload segment. Nov 24 05:12:53.903485471 (357h8m52.454955932s) [noise=true , address=45.91.116.6:4973 , piece_id=YFXRU76I3BSIAI3DKFRFERTVMJLR2CENQHYP3TMUCDBKBLWGGRYA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.903511801 (357h8m52.454982262s) [noise=true , address=77.90.31.14:4661 , piece_id=KLSYXSHBESAOQKDM7UIEJRBEU47LAZ4QLXQ7IIRHMGRHQDOTTGSQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.904736563 (357h8m52.456207024s) [noise=true , address=145.79.227.23:4547 , piece_id=77DA6ZG5K7FIM5VVFGLLIU3ZFPFSE6LYXA5ICFT32TGPTE4E6EWA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.905132177 (357h8m52.456602628s) [noise=true , address=145.79.227.6:4257 , piece_id=4YPKUHVNKCB25N5VNGRUAHIEAWOEIZL6FIM66XZGAI4SGNT6YV5Q, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.905529643 (357h8m52.457000104s) [noise=true , address=145.79.227.25:4619 , piece_id=Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906023413 (357h8m52.457493874s) [noise=true , address=145.79.227.22:4511 , piece_id=TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906454855 (357h8m52.457925306s) [noise=true , address=145.79.227.21:4474 , piece_id=3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906891877 (357h8m52.458362338s) [noise=true , address=145.79.227.10:4402 , piece_id=JT3O5M6VTUIFKBZKLTCBHTQWCOEYNWF4G2VW6DR73MVPIEAERLBQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.908246807 (357h8m52.459717268s) [noise=true , address=145.79.227.15:4875 , piece_id=VUFSGD36YPWJ32JILLC3ATEW2NYLIA4H3ROPTEFYXQQNXYPUC6VA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.908390811 (357h8m52.459861272s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.909633959 (357h8m52.46110442s) [noise=true , address=145.79.227.24:4582 , piece_id=EXSXMMBRJ4AGORCP4YYSET432YNOTBYOU32O42BZOO42LYOGBJJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.910221468 (357h8m52.461691929s) [noise=true , address=77.90.31.13:4628 , piece_id=TO7ZMNG64OQLA4AFEZKTSBPPB7DGLE72Q2DHZRWMGN54X2FQ47TA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.911006863 (357h8m52.462477315s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.911862011 (357h8m52.463332462s) [noise=true , address=45.91.116.4:4937 , piece_id=SL3XPUODGLVL5SNMZU7A6BT57IQ7GVGZE343ZJVNSFWC5TX3GVVQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.912556263 (357h8m52.464026714s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.913294699 (357h8m52.46476516s) [noise=true , address=45.91.116.11:4118 , piece_id=NEVUNIGHGP5AQPONBJ5J3IJCVJTUYGKHQNITKVG2ZABNTUWWZIBA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:55.574621520 (357h8m54.126091981s) [upload=314373]: Next returned nil segment Nov 24 05:12:55.574639704 (357h8m54.126110165s) [upload=314373]: Waiting for error group managing segments... Nov 24 05:12:55.966933719 (357h8m54.51840418s) [noise=true , address=77.90.31.14:4661 , piece_id=KLSYXSHBESAOQKDM7UIEJRBEU47LAZ4QLXQ7IIRHMGRHQDOTTGSQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:55.981186689 (357h8m54.53265714s) [noise=true , address=45.91.116.11:4118 , piece_id=NEVUNIGHGP5AQPONBJ5J3IJCVJTUYGKHQNITKVG2ZABNTUWWZIBA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.045206764 (357h8m54.596677225s) [noise=true , address=45.91.116.6:4973 , piece_id=YFXRU76I3BSIAI3DKFRFERTVMJLR2CENQHYP3TMUCDBKBLWGGRYA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.047372809 (357h8m54.59884327s) [noise=true , address=77.90.31.13:4628 , piece_id=TO7ZMNG64OQLA4AFEZKTSBPPB7DGLE72Q2DHZRWMGN54X2FQ47TA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.056652799 (357h8m54.60812326s) [noise=true , address=145.79.227.6:4257 , piece_id=4YPKUHVNKCB25N5VNGRUAHIEAWOEIZL6FIM66XZGAI4SGNT6YV5Q, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.059483058 (357h8m54.610953519s) [noise=true , address=145.79.227.21:4474 , piece_id=3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.070599767 (357h8m54.622070228s) [noise=true , address=145.79.227.10:4402 , piece_id=JT3O5M6VTUIFKBZKLTCBHTQWCOEYNWF4G2VW6DR73MVPIEAERLBQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.074296216 (357h8m54.625766677s) [noise=true , address=45.91.116.4:4937 , piece_id=SL3XPUODGLVL5SNMZU7A6BT57IQ7GVGZE343ZJVNSFWC5TX3GVVQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.081602177 (357h8m54.633072639s) [noise=true , address=145.79.227.22:4511 , piece_id=TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.082195698 (357h8m54.633666149s) [noise=true , address=145.79.227.23:4547 , piece_id=77DA6ZG5K7FIM5VVFGLLIU3ZFPFSE6LYXA5ICFT32TGPTE4E6EWA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.089421227 (357h8m54.640891688s) [noise=true , address=145.79.227.24:4582 , piece_id=EXSXMMBRJ4AGORCP4YYSET432YNOTBYOU32O42BZOO42LYOGBJJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.094493995 (357h8m54.645964446s) [noise=true , address=145.79.227.15:4875 , piece_id=VUFSGD36YPWJ32JILLC3ATEW2NYLIA4H3ROPTEFYXQQNXYPUC6VA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.111919777 (357h8m54.663390248s) [noise=true , address=145.79.227.25:4619 , piece_id=Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.111947239 (357h8m54.6634177s) [seg_pos={0 0}, upload=314373]: Segment reached optimal threshold of 13 pieces. Nov 24 05:12:56.112360226 (357h8m54.663830707s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112371026 (357h8m54.663841487s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112778422 (357h8m54.664248883s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112789903 (357h8m54.664260364s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112805814 (357h8m54.664276265s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112813649 (357h8m54.66428411s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112892599 (357h8m54.66436305s) [seg_pos={0 0}, upload=314373]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:12:56.113070117 (357h8m54.664540588s) [upload=314373]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:12:56.113079936 (357h8m54.664550387s) [upload=314373]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:12:56.609050819 (357h8m55.16052127s) [upload=314373]: Done uploading segments... Nov 24 05:12:56.609072030 (357h8m55.160542491s) [upload=314373]: Upload finished. err: <nil> Duration: 5.800712256s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X...X......] duration: 5.684541667s trace: https://jaeger.dev.storj.io/trace/514639791a8f826b FAIL (2025-11-24 05:17:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ) context deadline exceeded; piecestore: (Node ID: 12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, Piece ID: TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: 3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA) context deadline exceeded ======LOGS====== Nov 24 05:12:52.682445011 (357h8m51.233915462s) [upload=314373]: Starting upload Nov 24 05:12:52.682548268 (357h8m51.234018729s) [upload=314373]: Done starting upload Nov 24 05:12:52.682618251 (357h8m51.234088712s) [upload=314373]: Uploading segments... Nov 24 05:12:52.682764169 (357h8m51.23423466s) [upload=314373]: Got next segment. Inline: false Nov 24 05:12:52.682809646 (357h8m51.234280107s) [upload=314373]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:12:52.682816379 (357h8m51.23428684s) [upload=314373]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:12:53.903345875 (357h8m52.454816346s) [seg_pos={0 0}, upload=314373]: Begin upload segment... Nov 24 05:12:53.903444022 (357h8m52.454914523s) [seg_pos={0 0}, upload=314373]: Done begin upload segment. Nov 24 05:12:53.903485471 (357h8m52.454955932s) [noise=true , address=45.91.116.6:4973 , piece_id=YFXRU76I3BSIAI3DKFRFERTVMJLR2CENQHYP3TMUCDBKBLWGGRYA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.903511801 (357h8m52.454982262s) [noise=true , address=77.90.31.14:4661 , piece_id=KLSYXSHBESAOQKDM7UIEJRBEU47LAZ4QLXQ7IIRHMGRHQDOTTGSQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.904736563 (357h8m52.456207024s) [noise=true , address=145.79.227.23:4547 , piece_id=77DA6ZG5K7FIM5VVFGLLIU3ZFPFSE6LYXA5ICFT32TGPTE4E6EWA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.905132177 (357h8m52.456602628s) [noise=true , address=145.79.227.6:4257 , piece_id=4YPKUHVNKCB25N5VNGRUAHIEAWOEIZL6FIM66XZGAI4SGNT6YV5Q, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.905529643 (357h8m52.457000104s) [noise=true , address=145.79.227.25:4619 , piece_id=Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906023413 (357h8m52.457493874s) [noise=true , address=145.79.227.22:4511 , piece_id=TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906454855 (357h8m52.457925306s) [noise=true , address=145.79.227.21:4474 , piece_id=3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906891877 (357h8m52.458362338s) [noise=true , address=145.79.227.10:4402 , piece_id=JT3O5M6VTUIFKBZKLTCBHTQWCOEYNWF4G2VW6DR73MVPIEAERLBQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.908246807 (357h8m52.459717268s) [noise=true , address=145.79.227.15:4875 , piece_id=VUFSGD36YPWJ32JILLC3ATEW2NYLIA4H3ROPTEFYXQQNXYPUC6VA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.908390811 (357h8m52.459861272s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.909633959 (357h8m52.46110442s) [noise=true , address=145.79.227.24:4582 , piece_id=EXSXMMBRJ4AGORCP4YYSET432YNOTBYOU32O42BZOO42LYOGBJJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.910221468 (357h8m52.461691929s) [noise=true , address=77.90.31.13:4628 , piece_id=TO7ZMNG64OQLA4AFEZKTSBPPB7DGLE72Q2DHZRWMGN54X2FQ47TA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.911006863 (357h8m52.462477315s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.911862011 (357h8m52.463332462s) [noise=true , address=45.91.116.4:4937 , piece_id=SL3XPUODGLVL5SNMZU7A6BT57IQ7GVGZE343ZJVNSFWC5TX3GVVQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.912556263 (357h8m52.464026714s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.913294699 (357h8m52.46476516s) [noise=true , address=45.91.116.11:4118 , piece_id=NEVUNIGHGP5AQPONBJ5J3IJCVJTUYGKHQNITKVG2ZABNTUWWZIBA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:55.574621520 (357h8m54.126091981s) [upload=314373]: Next returned nil segment Nov 24 05:12:55.574639704 (357h8m54.126110165s) [upload=314373]: Waiting for error group managing segments... Nov 24 05:12:55.966933719 (357h8m54.51840418s) [noise=true , address=77.90.31.14:4661 , piece_id=KLSYXSHBESAOQKDM7UIEJRBEU47LAZ4QLXQ7IIRHMGRHQDOTTGSQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:55.981186689 (357h8m54.53265714s) [noise=true , address=45.91.116.11:4118 , piece_id=NEVUNIGHGP5AQPONBJ5J3IJCVJTUYGKHQNITKVG2ZABNTUWWZIBA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.045206764 (357h8m54.596677225s) [noise=true , address=45.91.116.6:4973 , piece_id=YFXRU76I3BSIAI3DKFRFERTVMJLR2CENQHYP3TMUCDBKBLWGGRYA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.047372809 (357h8m54.59884327s) [noise=true , address=77.90.31.13:4628 , piece_id=TO7ZMNG64OQLA4AFEZKTSBPPB7DGLE72Q2DHZRWMGN54X2FQ47TA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.056652799 (357h8m54.60812326s) [noise=true , address=145.79.227.6:4257 , piece_id=4YPKUHVNKCB25N5VNGRUAHIEAWOEIZL6FIM66XZGAI4SGNT6YV5Q, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.059483058 (357h8m54.610953519s) [noise=true , address=145.79.227.21:4474 , piece_id=3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.070599767 (357h8m54.622070228s) [noise=true , address=145.79.227.10:4402 , piece_id=JT3O5M6VTUIFKBZKLTCBHTQWCOEYNWF4G2VW6DR73MVPIEAERLBQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.074296216 (357h8m54.625766677s) [noise=true , address=45.91.116.4:4937 , piece_id=SL3XPUODGLVL5SNMZU7A6BT57IQ7GVGZE343ZJVNSFWC5TX3GVVQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.081602177 (357h8m54.633072639s) [noise=true , address=145.79.227.22:4511 , piece_id=TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.082195698 (357h8m54.633666149s) [noise=true , address=145.79.227.23:4547 , piece_id=77DA6ZG5K7FIM5VVFGLLIU3ZFPFSE6LYXA5ICFT32TGPTE4E6EWA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.089421227 (357h8m54.640891688s) [noise=true , address=145.79.227.24:4582 , piece_id=EXSXMMBRJ4AGORCP4YYSET432YNOTBYOU32O42BZOO42LYOGBJJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.094493995 (357h8m54.645964446s) [noise=true , address=145.79.227.15:4875 , piece_id=VUFSGD36YPWJ32JILLC3ATEW2NYLIA4H3ROPTEFYXQQNXYPUC6VA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.111919777 (357h8m54.663390248s) [noise=true , address=145.79.227.25:4619 , piece_id=Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.111947239 (357h8m54.6634177s) [seg_pos={0 0}, upload=314373]: Segment reached optimal threshold of 13 pieces. Nov 24 05:12:56.112360226 (357h8m54.663830707s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112371026 (357h8m54.663841487s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112778422 (357h8m54.664248883s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112789903 (357h8m54.664260364s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112805814 (357h8m54.664276265s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112813649 (357h8m54.66428411s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112892599 (357h8m54.66436305s) [seg_pos={0 0}, upload=314373]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:12:56.113070117 (357h8m54.664540588s) [upload=314373]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:12:56.113079936 (357h8m54.664550387s) [upload=314373]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:12:56.609050819 (357h8m55.16052127s) [upload=314373]: Done uploading segments... Nov 24 05:12:56.609072030 (357h8m55.160542491s) [upload=314373]: Upload finished. err: <nil> Duration: 5.684568518s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X...X.....] duration: 5.772635103s trace: https://jaeger.dev.storj.io/trace/514639791a8f826b FAIL (2025-11-24 05:16:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ) context deadline exceeded; piecestore: (Node ID: 12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, Piece ID: TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: 3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA) context deadline exceeded ======LOGS====== Nov 24 05:12:52.682445011 (357h8m51.233915462s) [upload=314373]: Starting upload Nov 24 05:12:52.682548268 (357h8m51.234018729s) [upload=314373]: Done starting upload Nov 24 05:12:52.682618251 (357h8m51.234088712s) [upload=314373]: Uploading segments... Nov 24 05:12:52.682764169 (357h8m51.23423466s) [upload=314373]: Got next segment. Inline: false Nov 24 05:12:52.682809646 (357h8m51.234280107s) [upload=314373]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:12:52.682816379 (357h8m51.23428684s) [upload=314373]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:12:53.903345875 (357h8m52.454816346s) [seg_pos={0 0}, upload=314373]: Begin upload segment... Nov 24 05:12:53.903444022 (357h8m52.454914523s) [seg_pos={0 0}, upload=314373]: Done begin upload segment. Nov 24 05:12:53.903485471 (357h8m52.454955932s) [noise=true , address=45.91.116.6:4973 , piece_id=YFXRU76I3BSIAI3DKFRFERTVMJLR2CENQHYP3TMUCDBKBLWGGRYA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.903511801 (357h8m52.454982262s) [noise=true , address=77.90.31.14:4661 , piece_id=KLSYXSHBESAOQKDM7UIEJRBEU47LAZ4QLXQ7IIRHMGRHQDOTTGSQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.904736563 (357h8m52.456207024s) [noise=true , address=145.79.227.23:4547 , piece_id=77DA6ZG5K7FIM5VVFGLLIU3ZFPFSE6LYXA5ICFT32TGPTE4E6EWA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.905132177 (357h8m52.456602628s) [noise=true , address=145.79.227.6:4257 , piece_id=4YPKUHVNKCB25N5VNGRUAHIEAWOEIZL6FIM66XZGAI4SGNT6YV5Q, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.905529643 (357h8m52.457000104s) [noise=true , address=145.79.227.25:4619 , piece_id=Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906023413 (357h8m52.457493874s) [noise=true , address=145.79.227.22:4511 , piece_id=TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906454855 (357h8m52.457925306s) [noise=true , address=145.79.227.21:4474 , piece_id=3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906891877 (357h8m52.458362338s) [noise=true , address=145.79.227.10:4402 , piece_id=JT3O5M6VTUIFKBZKLTCBHTQWCOEYNWF4G2VW6DR73MVPIEAERLBQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.908246807 (357h8m52.459717268s) [noise=true , address=145.79.227.15:4875 , piece_id=VUFSGD36YPWJ32JILLC3ATEW2NYLIA4H3ROPTEFYXQQNXYPUC6VA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.908390811 (357h8m52.459861272s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.909633959 (357h8m52.46110442s) [noise=true , address=145.79.227.24:4582 , piece_id=EXSXMMBRJ4AGORCP4YYSET432YNOTBYOU32O42BZOO42LYOGBJJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.910221468 (357h8m52.461691929s) [noise=true , address=77.90.31.13:4628 , piece_id=TO7ZMNG64OQLA4AFEZKTSBPPB7DGLE72Q2DHZRWMGN54X2FQ47TA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.911006863 (357h8m52.462477315s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.911862011 (357h8m52.463332462s) [noise=true , address=45.91.116.4:4937 , piece_id=SL3XPUODGLVL5SNMZU7A6BT57IQ7GVGZE343ZJVNSFWC5TX3GVVQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.912556263 (357h8m52.464026714s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.913294699 (357h8m52.46476516s) [noise=true , address=45.91.116.11:4118 , piece_id=NEVUNIGHGP5AQPONBJ5J3IJCVJTUYGKHQNITKVG2ZABNTUWWZIBA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:55.574621520 (357h8m54.126091981s) [upload=314373]: Next returned nil segment Nov 24 05:12:55.574639704 (357h8m54.126110165s) [upload=314373]: Waiting for error group managing segments... Nov 24 05:12:55.966933719 (357h8m54.51840418s) [noise=true , address=77.90.31.14:4661 , piece_id=KLSYXSHBESAOQKDM7UIEJRBEU47LAZ4QLXQ7IIRHMGRHQDOTTGSQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:55.981186689 (357h8m54.53265714s) [noise=true , address=45.91.116.11:4118 , piece_id=NEVUNIGHGP5AQPONBJ5J3IJCVJTUYGKHQNITKVG2ZABNTUWWZIBA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.045206764 (357h8m54.596677225s) [noise=true , address=45.91.116.6:4973 , piece_id=YFXRU76I3BSIAI3DKFRFERTVMJLR2CENQHYP3TMUCDBKBLWGGRYA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.047372809 (357h8m54.59884327s) [noise=true , address=77.90.31.13:4628 , piece_id=TO7ZMNG64OQLA4AFEZKTSBPPB7DGLE72Q2DHZRWMGN54X2FQ47TA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.056652799 (357h8m54.60812326s) [noise=true , address=145.79.227.6:4257 , piece_id=4YPKUHVNKCB25N5VNGRUAHIEAWOEIZL6FIM66XZGAI4SGNT6YV5Q, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.059483058 (357h8m54.610953519s) [noise=true , address=145.79.227.21:4474 , piece_id=3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.070599767 (357h8m54.622070228s) [noise=true , address=145.79.227.10:4402 , piece_id=JT3O5M6VTUIFKBZKLTCBHTQWCOEYNWF4G2VW6DR73MVPIEAERLBQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.074296216 (357h8m54.625766677s) [noise=true , address=45.91.116.4:4937 , piece_id=SL3XPUODGLVL5SNMZU7A6BT57IQ7GVGZE343ZJVNSFWC5TX3GVVQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.081602177 (357h8m54.633072639s) [noise=true , address=145.79.227.22:4511 , piece_id=TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.082195698 (357h8m54.633666149s) [noise=true , address=145.79.227.23:4547 , piece_id=77DA6ZG5K7FIM5VVFGLLIU3ZFPFSE6LYXA5ICFT32TGPTE4E6EWA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.089421227 (357h8m54.640891688s) [noise=true , address=145.79.227.24:4582 , piece_id=EXSXMMBRJ4AGORCP4YYSET432YNOTBYOU32O42BZOO42LYOGBJJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.094493995 (357h8m54.645964446s) [noise=true , address=145.79.227.15:4875 , piece_id=VUFSGD36YPWJ32JILLC3ATEW2NYLIA4H3ROPTEFYXQQNXYPUC6VA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.111919777 (357h8m54.663390248s) [noise=true , address=145.79.227.25:4619 , piece_id=Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.111947239 (357h8m54.6634177s) [seg_pos={0 0}, upload=314373]: Segment reached optimal threshold of 13 pieces. Nov 24 05:12:56.112360226 (357h8m54.663830707s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112371026 (357h8m54.663841487s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112778422 (357h8m54.664248883s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112789903 (357h8m54.664260364s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112805814 (357h8m54.664276265s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112813649 (357h8m54.66428411s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112892599 (357h8m54.66436305s) [seg_pos={0 0}, upload=314373]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:12:56.113070117 (357h8m54.664540588s) [upload=314373]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:12:56.113079936 (357h8m54.664550387s) [upload=314373]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:12:56.609050819 (357h8m55.16052127s) [upload=314373]: Done uploading segments... Nov 24 05:12:56.609072030 (357h8m55.160542491s) [upload=314373]: Upload finished. err: <nil> Duration: 5.772694205s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X...X....] duration: 5.631387963s trace: https://jaeger.dev.storj.io/trace/514639791a8f826b FAIL (2025-11-24 05:15:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ) context deadline exceeded; piecestore: (Node ID: 12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, Piece ID: TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: 3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA) context deadline exceeded ======LOGS====== Nov 24 05:12:52.682445011 (357h8m51.233915462s) [upload=314373]: Starting upload Nov 24 05:12:52.682548268 (357h8m51.234018729s) [upload=314373]: Done starting upload Nov 24 05:12:52.682618251 (357h8m51.234088712s) [upload=314373]: Uploading segments... Nov 24 05:12:52.682764169 (357h8m51.23423466s) [upload=314373]: Got next segment. Inline: false Nov 24 05:12:52.682809646 (357h8m51.234280107s) [upload=314373]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:12:52.682816379 (357h8m51.23428684s) [upload=314373]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:12:53.903345875 (357h8m52.454816346s) [seg_pos={0 0}, upload=314373]: Begin upload segment... Nov 24 05:12:53.903444022 (357h8m52.454914523s) [seg_pos={0 0}, upload=314373]: Done begin upload segment. Nov 24 05:12:53.903485471 (357h8m52.454955932s) [noise=true , address=45.91.116.6:4973 , piece_id=YFXRU76I3BSIAI3DKFRFERTVMJLR2CENQHYP3TMUCDBKBLWGGRYA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.903511801 (357h8m52.454982262s) [noise=true , address=77.90.31.14:4661 , piece_id=KLSYXSHBESAOQKDM7UIEJRBEU47LAZ4QLXQ7IIRHMGRHQDOTTGSQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.904736563 (357h8m52.456207024s) [noise=true , address=145.79.227.23:4547 , piece_id=77DA6ZG5K7FIM5VVFGLLIU3ZFPFSE6LYXA5ICFT32TGPTE4E6EWA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.905132177 (357h8m52.456602628s) [noise=true , address=145.79.227.6:4257 , piece_id=4YPKUHVNKCB25N5VNGRUAHIEAWOEIZL6FIM66XZGAI4SGNT6YV5Q, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.905529643 (357h8m52.457000104s) [noise=true , address=145.79.227.25:4619 , piece_id=Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906023413 (357h8m52.457493874s) [noise=true , address=145.79.227.22:4511 , piece_id=TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906454855 (357h8m52.457925306s) [noise=true , address=145.79.227.21:4474 , piece_id=3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906891877 (357h8m52.458362338s) [noise=true , address=145.79.227.10:4402 , piece_id=JT3O5M6VTUIFKBZKLTCBHTQWCOEYNWF4G2VW6DR73MVPIEAERLBQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.908246807 (357h8m52.459717268s) [noise=true , address=145.79.227.15:4875 , piece_id=VUFSGD36YPWJ32JILLC3ATEW2NYLIA4H3ROPTEFYXQQNXYPUC6VA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.908390811 (357h8m52.459861272s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.909633959 (357h8m52.46110442s) [noise=true , address=145.79.227.24:4582 , piece_id=EXSXMMBRJ4AGORCP4YYSET432YNOTBYOU32O42BZOO42LYOGBJJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.910221468 (357h8m52.461691929s) [noise=true , address=77.90.31.13:4628 , piece_id=TO7ZMNG64OQLA4AFEZKTSBPPB7DGLE72Q2DHZRWMGN54X2FQ47TA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.911006863 (357h8m52.462477315s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.911862011 (357h8m52.463332462s) [noise=true , address=45.91.116.4:4937 , piece_id=SL3XPUODGLVL5SNMZU7A6BT57IQ7GVGZE343ZJVNSFWC5TX3GVVQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.912556263 (357h8m52.464026714s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.913294699 (357h8m52.46476516s) [noise=true , address=45.91.116.11:4118 , piece_id=NEVUNIGHGP5AQPONBJ5J3IJCVJTUYGKHQNITKVG2ZABNTUWWZIBA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:55.574621520 (357h8m54.126091981s) [upload=314373]: Next returned nil segment Nov 24 05:12:55.574639704 (357h8m54.126110165s) [upload=314373]: Waiting for error group managing segments... Nov 24 05:12:55.966933719 (357h8m54.51840418s) [noise=true , address=77.90.31.14:4661 , piece_id=KLSYXSHBESAOQKDM7UIEJRBEU47LAZ4QLXQ7IIRHMGRHQDOTTGSQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:55.981186689 (357h8m54.53265714s) [noise=true , address=45.91.116.11:4118 , piece_id=NEVUNIGHGP5AQPONBJ5J3IJCVJTUYGKHQNITKVG2ZABNTUWWZIBA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.045206764 (357h8m54.596677225s) [noise=true , address=45.91.116.6:4973 , piece_id=YFXRU76I3BSIAI3DKFRFERTVMJLR2CENQHYP3TMUCDBKBLWGGRYA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.047372809 (357h8m54.59884327s) [noise=true , address=77.90.31.13:4628 , piece_id=TO7ZMNG64OQLA4AFEZKTSBPPB7DGLE72Q2DHZRWMGN54X2FQ47TA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.056652799 (357h8m54.60812326s) [noise=true , address=145.79.227.6:4257 , piece_id=4YPKUHVNKCB25N5VNGRUAHIEAWOEIZL6FIM66XZGAI4SGNT6YV5Q, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.059483058 (357h8m54.610953519s) [noise=true , address=145.79.227.21:4474 , piece_id=3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.070599767 (357h8m54.622070228s) [noise=true , address=145.79.227.10:4402 , piece_id=JT3O5M6VTUIFKBZKLTCBHTQWCOEYNWF4G2VW6DR73MVPIEAERLBQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.074296216 (357h8m54.625766677s) [noise=true , address=45.91.116.4:4937 , piece_id=SL3XPUODGLVL5SNMZU7A6BT57IQ7GVGZE343ZJVNSFWC5TX3GVVQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.081602177 (357h8m54.633072639s) [noise=true , address=145.79.227.22:4511 , piece_id=TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.082195698 (357h8m54.633666149s) [noise=true , address=145.79.227.23:4547 , piece_id=77DA6ZG5K7FIM5VVFGLLIU3ZFPFSE6LYXA5ICFT32TGPTE4E6EWA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.089421227 (357h8m54.640891688s) [noise=true , address=145.79.227.24:4582 , piece_id=EXSXMMBRJ4AGORCP4YYSET432YNOTBYOU32O42BZOO42LYOGBJJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.094493995 (357h8m54.645964446s) [noise=true , address=145.79.227.15:4875 , piece_id=VUFSGD36YPWJ32JILLC3ATEW2NYLIA4H3ROPTEFYXQQNXYPUC6VA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.111919777 (357h8m54.663390248s) [noise=true , address=145.79.227.25:4619 , piece_id=Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.111947239 (357h8m54.6634177s) [seg_pos={0 0}, upload=314373]: Segment reached optimal threshold of 13 pieces. Nov 24 05:12:56.112360226 (357h8m54.663830707s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112371026 (357h8m54.663841487s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112778422 (357h8m54.664248883s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112789903 (357h8m54.664260364s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112805814 (357h8m54.664276265s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112813649 (357h8m54.66428411s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112892599 (357h8m54.66436305s) [seg_pos={0 0}, upload=314373]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:12:56.113070117 (357h8m54.664540588s) [upload=314373]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:12:56.113079936 (357h8m54.664550387s) [upload=314373]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:12:56.609050819 (357h8m55.16052127s) [upload=314373]: Done uploading segments... Nov 24 05:12:56.609072030 (357h8m55.160542491s) [upload=314373]: Upload finished. err: <nil> Duration: 5.631424612s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X...X...] duration: 6.040408629s trace: https://jaeger.dev.storj.io/trace/514639791a8f826b FAIL (2025-11-24 05:15:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ) context deadline exceeded; piecestore: (Node ID: 12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, Piece ID: TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: 3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA) context deadline exceeded ======LOGS====== Nov 24 05:12:52.682445011 (357h8m51.233915462s) [upload=314373]: Starting upload Nov 24 05:12:52.682548268 (357h8m51.234018729s) [upload=314373]: Done starting upload Nov 24 05:12:52.682618251 (357h8m51.234088712s) [upload=314373]: Uploading segments... Nov 24 05:12:52.682764169 (357h8m51.23423466s) [upload=314373]: Got next segment. Inline: false Nov 24 05:12:52.682809646 (357h8m51.234280107s) [upload=314373]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:12:52.682816379 (357h8m51.23428684s) [upload=314373]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:12:53.903345875 (357h8m52.454816346s) [seg_pos={0 0}, upload=314373]: Begin upload segment... Nov 24 05:12:53.903444022 (357h8m52.454914523s) [seg_pos={0 0}, upload=314373]: Done begin upload segment. Nov 24 05:12:53.903485471 (357h8m52.454955932s) [noise=true , address=45.91.116.6:4973 , piece_id=YFXRU76I3BSIAI3DKFRFERTVMJLR2CENQHYP3TMUCDBKBLWGGRYA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.903511801 (357h8m52.454982262s) [noise=true , address=77.90.31.14:4661 , piece_id=KLSYXSHBESAOQKDM7UIEJRBEU47LAZ4QLXQ7IIRHMGRHQDOTTGSQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.904736563 (357h8m52.456207024s) [noise=true , address=145.79.227.23:4547 , piece_id=77DA6ZG5K7FIM5VVFGLLIU3ZFPFSE6LYXA5ICFT32TGPTE4E6EWA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.905132177 (357h8m52.456602628s) [noise=true , address=145.79.227.6:4257 , piece_id=4YPKUHVNKCB25N5VNGRUAHIEAWOEIZL6FIM66XZGAI4SGNT6YV5Q, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.905529643 (357h8m52.457000104s) [noise=true , address=145.79.227.25:4619 , piece_id=Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906023413 (357h8m52.457493874s) [noise=true , address=145.79.227.22:4511 , piece_id=TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906454855 (357h8m52.457925306s) [noise=true , address=145.79.227.21:4474 , piece_id=3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906891877 (357h8m52.458362338s) [noise=true , address=145.79.227.10:4402 , piece_id=JT3O5M6VTUIFKBZKLTCBHTQWCOEYNWF4G2VW6DR73MVPIEAERLBQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.908246807 (357h8m52.459717268s) [noise=true , address=145.79.227.15:4875 , piece_id=VUFSGD36YPWJ32JILLC3ATEW2NYLIA4H3ROPTEFYXQQNXYPUC6VA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.908390811 (357h8m52.459861272s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.909633959 (357h8m52.46110442s) [noise=true , address=145.79.227.24:4582 , piece_id=EXSXMMBRJ4AGORCP4YYSET432YNOTBYOU32O42BZOO42LYOGBJJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.910221468 (357h8m52.461691929s) [noise=true , address=77.90.31.13:4628 , piece_id=TO7ZMNG64OQLA4AFEZKTSBPPB7DGLE72Q2DHZRWMGN54X2FQ47TA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.911006863 (357h8m52.462477315s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.911862011 (357h8m52.463332462s) [noise=true , address=45.91.116.4:4937 , piece_id=SL3XPUODGLVL5SNMZU7A6BT57IQ7GVGZE343ZJVNSFWC5TX3GVVQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.912556263 (357h8m52.464026714s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.913294699 (357h8m52.46476516s) [noise=true , address=45.91.116.11:4118 , piece_id=NEVUNIGHGP5AQPONBJ5J3IJCVJTUYGKHQNITKVG2ZABNTUWWZIBA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:55.574621520 (357h8m54.126091981s) [upload=314373]: Next returned nil segment Nov 24 05:12:55.574639704 (357h8m54.126110165s) [upload=314373]: Waiting for error group managing segments... Nov 24 05:12:55.966933719 (357h8m54.51840418s) [noise=true , address=77.90.31.14:4661 , piece_id=KLSYXSHBESAOQKDM7UIEJRBEU47LAZ4QLXQ7IIRHMGRHQDOTTGSQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:55.981186689 (357h8m54.53265714s) [noise=true , address=45.91.116.11:4118 , piece_id=NEVUNIGHGP5AQPONBJ5J3IJCVJTUYGKHQNITKVG2ZABNTUWWZIBA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.045206764 (357h8m54.596677225s) [noise=true , address=45.91.116.6:4973 , piece_id=YFXRU76I3BSIAI3DKFRFERTVMJLR2CENQHYP3TMUCDBKBLWGGRYA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.047372809 (357h8m54.59884327s) [noise=true , address=77.90.31.13:4628 , piece_id=TO7ZMNG64OQLA4AFEZKTSBPPB7DGLE72Q2DHZRWMGN54X2FQ47TA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.056652799 (357h8m54.60812326s) [noise=true , address=145.79.227.6:4257 , piece_id=4YPKUHVNKCB25N5VNGRUAHIEAWOEIZL6FIM66XZGAI4SGNT6YV5Q, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.059483058 (357h8m54.610953519s) [noise=true , address=145.79.227.21:4474 , piece_id=3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.070599767 (357h8m54.622070228s) [noise=true , address=145.79.227.10:4402 , piece_id=JT3O5M6VTUIFKBZKLTCBHTQWCOEYNWF4G2VW6DR73MVPIEAERLBQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.074296216 (357h8m54.625766677s) [noise=true , address=45.91.116.4:4937 , piece_id=SL3XPUODGLVL5SNMZU7A6BT57IQ7GVGZE343ZJVNSFWC5TX3GVVQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.081602177 (357h8m54.633072639s) [noise=true , address=145.79.227.22:4511 , piece_id=TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.082195698 (357h8m54.633666149s) [noise=true , address=145.79.227.23:4547 , piece_id=77DA6ZG5K7FIM5VVFGLLIU3ZFPFSE6LYXA5ICFT32TGPTE4E6EWA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.089421227 (357h8m54.640891688s) [noise=true , address=145.79.227.24:4582 , piece_id=EXSXMMBRJ4AGORCP4YYSET432YNOTBYOU32O42BZOO42LYOGBJJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.094493995 (357h8m54.645964446s) [noise=true , address=145.79.227.15:4875 , piece_id=VUFSGD36YPWJ32JILLC3ATEW2NYLIA4H3ROPTEFYXQQNXYPUC6VA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.111919777 (357h8m54.663390248s) [noise=true , address=145.79.227.25:4619 , piece_id=Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.111947239 (357h8m54.6634177s) [seg_pos={0 0}, upload=314373]: Segment reached optimal threshold of 13 pieces. Nov 24 05:12:56.112360226 (357h8m54.663830707s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112371026 (357h8m54.663841487s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112778422 (357h8m54.664248883s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112789903 (357h8m54.664260364s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112805814 (357h8m54.664276265s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112813649 (357h8m54.66428411s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112892599 (357h8m54.66436305s) [seg_pos={0 0}, upload=314373]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:12:56.113070117 (357h8m54.664540588s) [upload=314373]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:12:56.113079936 (357h8m54.664550387s) [upload=314373]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:12:56.609050819 (357h8m55.16052127s) [upload=314373]: Done uploading segments... Nov 24 05:12:56.609072030 (357h8m55.160542491s) [upload=314373]: Upload finished. err: <nil> Duration: 6.040438997s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X...X..] duration: 5.846271607s trace: https://jaeger.dev.storj.io/trace/514639791a8f826b FAIL (2025-11-24 05:14:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ) context deadline exceeded; piecestore: (Node ID: 12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, Piece ID: TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: 3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA) context deadline exceeded ======LOGS====== Nov 24 05:12:52.682445011 (357h8m51.233915462s) [upload=314373]: Starting upload Nov 24 05:12:52.682548268 (357h8m51.234018729s) [upload=314373]: Done starting upload Nov 24 05:12:52.682618251 (357h8m51.234088712s) [upload=314373]: Uploading segments... Nov 24 05:12:52.682764169 (357h8m51.23423466s) [upload=314373]: Got next segment. Inline: false Nov 24 05:12:52.682809646 (357h8m51.234280107s) [upload=314373]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:12:52.682816379 (357h8m51.23428684s) [upload=314373]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:12:53.903345875 (357h8m52.454816346s) [seg_pos={0 0}, upload=314373]: Begin upload segment... Nov 24 05:12:53.903444022 (357h8m52.454914523s) [seg_pos={0 0}, upload=314373]: Done begin upload segment. Nov 24 05:12:53.903485471 (357h8m52.454955932s) [noise=true , address=45.91.116.6:4973 , piece_id=YFXRU76I3BSIAI3DKFRFERTVMJLR2CENQHYP3TMUCDBKBLWGGRYA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.903511801 (357h8m52.454982262s) [noise=true , address=77.90.31.14:4661 , piece_id=KLSYXSHBESAOQKDM7UIEJRBEU47LAZ4QLXQ7IIRHMGRHQDOTTGSQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.904736563 (357h8m52.456207024s) [noise=true , address=145.79.227.23:4547 , piece_id=77DA6ZG5K7FIM5VVFGLLIU3ZFPFSE6LYXA5ICFT32TGPTE4E6EWA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.905132177 (357h8m52.456602628s) [noise=true , address=145.79.227.6:4257 , piece_id=4YPKUHVNKCB25N5VNGRUAHIEAWOEIZL6FIM66XZGAI4SGNT6YV5Q, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.905529643 (357h8m52.457000104s) [noise=true , address=145.79.227.25:4619 , piece_id=Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906023413 (357h8m52.457493874s) [noise=true , address=145.79.227.22:4511 , piece_id=TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906454855 (357h8m52.457925306s) [noise=true , address=145.79.227.21:4474 , piece_id=3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906891877 (357h8m52.458362338s) [noise=true , address=145.79.227.10:4402 , piece_id=JT3O5M6VTUIFKBZKLTCBHTQWCOEYNWF4G2VW6DR73MVPIEAERLBQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.908246807 (357h8m52.459717268s) [noise=true , address=145.79.227.15:4875 , piece_id=VUFSGD36YPWJ32JILLC3ATEW2NYLIA4H3ROPTEFYXQQNXYPUC6VA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.908390811 (357h8m52.459861272s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.909633959 (357h8m52.46110442s) [noise=true , address=145.79.227.24:4582 , piece_id=EXSXMMBRJ4AGORCP4YYSET432YNOTBYOU32O42BZOO42LYOGBJJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.910221468 (357h8m52.461691929s) [noise=true , address=77.90.31.13:4628 , piece_id=TO7ZMNG64OQLA4AFEZKTSBPPB7DGLE72Q2DHZRWMGN54X2FQ47TA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.911006863 (357h8m52.462477315s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.911862011 (357h8m52.463332462s) [noise=true , address=45.91.116.4:4937 , piece_id=SL3XPUODGLVL5SNMZU7A6BT57IQ7GVGZE343ZJVNSFWC5TX3GVVQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.912556263 (357h8m52.464026714s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.913294699 (357h8m52.46476516s) [noise=true , address=45.91.116.11:4118 , piece_id=NEVUNIGHGP5AQPONBJ5J3IJCVJTUYGKHQNITKVG2ZABNTUWWZIBA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:55.574621520 (357h8m54.126091981s) [upload=314373]: Next returned nil segment Nov 24 05:12:55.574639704 (357h8m54.126110165s) [upload=314373]: Waiting for error group managing segments... Nov 24 05:12:55.966933719 (357h8m54.51840418s) [noise=true , address=77.90.31.14:4661 , piece_id=KLSYXSHBESAOQKDM7UIEJRBEU47LAZ4QLXQ7IIRHMGRHQDOTTGSQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:55.981186689 (357h8m54.53265714s) [noise=true , address=45.91.116.11:4118 , piece_id=NEVUNIGHGP5AQPONBJ5J3IJCVJTUYGKHQNITKVG2ZABNTUWWZIBA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.045206764 (357h8m54.596677225s) [noise=true , address=45.91.116.6:4973 , piece_id=YFXRU76I3BSIAI3DKFRFERTVMJLR2CENQHYP3TMUCDBKBLWGGRYA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.047372809 (357h8m54.59884327s) [noise=true , address=77.90.31.13:4628 , piece_id=TO7ZMNG64OQLA4AFEZKTSBPPB7DGLE72Q2DHZRWMGN54X2FQ47TA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.056652799 (357h8m54.60812326s) [noise=true , address=145.79.227.6:4257 , piece_id=4YPKUHVNKCB25N5VNGRUAHIEAWOEIZL6FIM66XZGAI4SGNT6YV5Q, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.059483058 (357h8m54.610953519s) [noise=true , address=145.79.227.21:4474 , piece_id=3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.070599767 (357h8m54.622070228s) [noise=true , address=145.79.227.10:4402 , piece_id=JT3O5M6VTUIFKBZKLTCBHTQWCOEYNWF4G2VW6DR73MVPIEAERLBQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.074296216 (357h8m54.625766677s) [noise=true , address=45.91.116.4:4937 , piece_id=SL3XPUODGLVL5SNMZU7A6BT57IQ7GVGZE343ZJVNSFWC5TX3GVVQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.081602177 (357h8m54.633072639s) [noise=true , address=145.79.227.22:4511 , piece_id=TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.082195698 (357h8m54.633666149s) [noise=true , address=145.79.227.23:4547 , piece_id=77DA6ZG5K7FIM5VVFGLLIU3ZFPFSE6LYXA5ICFT32TGPTE4E6EWA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.089421227 (357h8m54.640891688s) [noise=true , address=145.79.227.24:4582 , piece_id=EXSXMMBRJ4AGORCP4YYSET432YNOTBYOU32O42BZOO42LYOGBJJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.094493995 (357h8m54.645964446s) [noise=true , address=145.79.227.15:4875 , piece_id=VUFSGD36YPWJ32JILLC3ATEW2NYLIA4H3ROPTEFYXQQNXYPUC6VA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.111919777 (357h8m54.663390248s) [noise=true , address=145.79.227.25:4619 , piece_id=Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.111947239 (357h8m54.6634177s) [seg_pos={0 0}, upload=314373]: Segment reached optimal threshold of 13 pieces. Nov 24 05:12:56.112360226 (357h8m54.663830707s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112371026 (357h8m54.663841487s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112778422 (357h8m54.664248883s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112789903 (357h8m54.664260364s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112805814 (357h8m54.664276265s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112813649 (357h8m54.66428411s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112892599 (357h8m54.66436305s) [seg_pos={0 0}, upload=314373]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:12:56.113070117 (357h8m54.664540588s) [upload=314373]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:12:56.113079936 (357h8m54.664550387s) [upload=314373]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:12:56.609050819 (357h8m55.16052127s) [upload=314373]: Done uploading segments... Nov 24 05:12:56.609072030 (357h8m55.160542491s) [upload=314373]: Upload finished. err: <nil> Duration: 5.846304842s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X...X.] duration: 25.849002077s trace: https://jaeger.dev.storj.io/trace/514639791a8f826b FAIL (2025-11-24 05:13:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ) context deadline exceeded; piecestore: (Node ID: 12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, Piece ID: TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: 3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA) context deadline exceeded ======LOGS====== Nov 24 05:12:52.682445011 (357h8m51.233915462s) [upload=314373]: Starting upload Nov 24 05:12:52.682548268 (357h8m51.234018729s) [upload=314373]: Done starting upload Nov 24 05:12:52.682618251 (357h8m51.234088712s) [upload=314373]: Uploading segments... Nov 24 05:12:52.682764169 (357h8m51.23423466s) [upload=314373]: Got next segment. Inline: false Nov 24 05:12:52.682809646 (357h8m51.234280107s) [upload=314373]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:12:52.682816379 (357h8m51.23428684s) [upload=314373]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:12:53.903345875 (357h8m52.454816346s) [seg_pos={0 0}, upload=314373]: Begin upload segment... Nov 24 05:12:53.903444022 (357h8m52.454914523s) [seg_pos={0 0}, upload=314373]: Done begin upload segment. Nov 24 05:12:53.903485471 (357h8m52.454955932s) [noise=true , address=45.91.116.6:4973 , piece_id=YFXRU76I3BSIAI3DKFRFERTVMJLR2CENQHYP3TMUCDBKBLWGGRYA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.903511801 (357h8m52.454982262s) [noise=true , address=77.90.31.14:4661 , piece_id=KLSYXSHBESAOQKDM7UIEJRBEU47LAZ4QLXQ7IIRHMGRHQDOTTGSQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.904736563 (357h8m52.456207024s) [noise=true , address=145.79.227.23:4547 , piece_id=77DA6ZG5K7FIM5VVFGLLIU3ZFPFSE6LYXA5ICFT32TGPTE4E6EWA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.905132177 (357h8m52.456602628s) [noise=true , address=145.79.227.6:4257 , piece_id=4YPKUHVNKCB25N5VNGRUAHIEAWOEIZL6FIM66XZGAI4SGNT6YV5Q, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.905529643 (357h8m52.457000104s) [noise=true , address=145.79.227.25:4619 , piece_id=Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906023413 (357h8m52.457493874s) [noise=true , address=145.79.227.22:4511 , piece_id=TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906454855 (357h8m52.457925306s) [noise=true , address=145.79.227.21:4474 , piece_id=3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906891877 (357h8m52.458362338s) [noise=true , address=145.79.227.10:4402 , piece_id=JT3O5M6VTUIFKBZKLTCBHTQWCOEYNWF4G2VW6DR73MVPIEAERLBQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.908246807 (357h8m52.459717268s) [noise=true , address=145.79.227.15:4875 , piece_id=VUFSGD36YPWJ32JILLC3ATEW2NYLIA4H3ROPTEFYXQQNXYPUC6VA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.908390811 (357h8m52.459861272s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.909633959 (357h8m52.46110442s) [noise=true , address=145.79.227.24:4582 , piece_id=EXSXMMBRJ4AGORCP4YYSET432YNOTBYOU32O42BZOO42LYOGBJJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.910221468 (357h8m52.461691929s) [noise=true , address=77.90.31.13:4628 , piece_id=TO7ZMNG64OQLA4AFEZKTSBPPB7DGLE72Q2DHZRWMGN54X2FQ47TA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.911006863 (357h8m52.462477315s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.911862011 (357h8m52.463332462s) [noise=true , address=45.91.116.4:4937 , piece_id=SL3XPUODGLVL5SNMZU7A6BT57IQ7GVGZE343ZJVNSFWC5TX3GVVQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.912556263 (357h8m52.464026714s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.913294699 (357h8m52.46476516s) [noise=true , address=45.91.116.11:4118 , piece_id=NEVUNIGHGP5AQPONBJ5J3IJCVJTUYGKHQNITKVG2ZABNTUWWZIBA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:55.574621520 (357h8m54.126091981s) [upload=314373]: Next returned nil segment Nov 24 05:12:55.574639704 (357h8m54.126110165s) [upload=314373]: Waiting for error group managing segments... Nov 24 05:12:55.966933719 (357h8m54.51840418s) [noise=true , address=77.90.31.14:4661 , piece_id=KLSYXSHBESAOQKDM7UIEJRBEU47LAZ4QLXQ7IIRHMGRHQDOTTGSQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:55.981186689 (357h8m54.53265714s) [noise=true , address=45.91.116.11:4118 , piece_id=NEVUNIGHGP5AQPONBJ5J3IJCVJTUYGKHQNITKVG2ZABNTUWWZIBA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.045206764 (357h8m54.596677225s) [noise=true , address=45.91.116.6:4973 , piece_id=YFXRU76I3BSIAI3DKFRFERTVMJLR2CENQHYP3TMUCDBKBLWGGRYA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.047372809 (357h8m54.59884327s) [noise=true , address=77.90.31.13:4628 , piece_id=TO7ZMNG64OQLA4AFEZKTSBPPB7DGLE72Q2DHZRWMGN54X2FQ47TA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.056652799 (357h8m54.60812326s) [noise=true , address=145.79.227.6:4257 , piece_id=4YPKUHVNKCB25N5VNGRUAHIEAWOEIZL6FIM66XZGAI4SGNT6YV5Q, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.059483058 (357h8m54.610953519s) [noise=true , address=145.79.227.21:4474 , piece_id=3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.070599767 (357h8m54.622070228s) [noise=true , address=145.79.227.10:4402 , piece_id=JT3O5M6VTUIFKBZKLTCBHTQWCOEYNWF4G2VW6DR73MVPIEAERLBQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.074296216 (357h8m54.625766677s) [noise=true , address=45.91.116.4:4937 , piece_id=SL3XPUODGLVL5SNMZU7A6BT57IQ7GVGZE343ZJVNSFWC5TX3GVVQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.081602177 (357h8m54.633072639s) [noise=true , address=145.79.227.22:4511 , piece_id=TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.082195698 (357h8m54.633666149s) [noise=true , address=145.79.227.23:4547 , piece_id=77DA6ZG5K7FIM5VVFGLLIU3ZFPFSE6LYXA5ICFT32TGPTE4E6EWA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.089421227 (357h8m54.640891688s) [noise=true , address=145.79.227.24:4582 , piece_id=EXSXMMBRJ4AGORCP4YYSET432YNOTBYOU32O42BZOO42LYOGBJJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.094493995 (357h8m54.645964446s) [noise=true , address=145.79.227.15:4875 , piece_id=VUFSGD36YPWJ32JILLC3ATEW2NYLIA4H3ROPTEFYXQQNXYPUC6VA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.111919777 (357h8m54.663390248s) [noise=true , address=145.79.227.25:4619 , piece_id=Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.111947239 (357h8m54.6634177s) [seg_pos={0 0}, upload=314373]: Segment reached optimal threshold of 13 pieces. Nov 24 05:12:56.112360226 (357h8m54.663830707s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112371026 (357h8m54.663841487s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112778422 (357h8m54.664248883s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112789903 (357h8m54.664260364s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112805814 (357h8m54.664276265s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112813649 (357h8m54.66428411s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112892599 (357h8m54.66436305s) [seg_pos={0 0}, upload=314373]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:12:56.113070117 (357h8m54.664540588s) [upload=314373]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:12:56.113079936 (357h8m54.664550387s) [upload=314373]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:12:56.609050819 (357h8m55.16052127s) [upload=314373]: Done uploading segments... Nov 24 05:12:56.609072030 (357h8m55.160542491s) [upload=314373]: Upload finished. err: <nil> Duration: 25.849098331s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X...X] duration: 30.004536722s trace: https://jaeger.dev.storj.io/trace/514639791a8f826b FAIL (2025-11-24 05:13:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ) context deadline exceeded; piecestore: (Node ID: 12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, Piece ID: TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: 3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA) context deadline exceeded ======LOGS====== Nov 24 05:12:52.682445011 (357h8m51.233915462s) [upload=314373]: Starting upload Nov 24 05:12:52.682548268 (357h8m51.234018729s) [upload=314373]: Done starting upload Nov 24 05:12:52.682618251 (357h8m51.234088712s) [upload=314373]: Uploading segments... Nov 24 05:12:52.682764169 (357h8m51.23423466s) [upload=314373]: Got next segment. Inline: false Nov 24 05:12:52.682809646 (357h8m51.234280107s) [upload=314373]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:12:52.682816379 (357h8m51.23428684s) [upload=314373]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:12:53.903345875 (357h8m52.454816346s) [seg_pos={0 0}, upload=314373]: Begin upload segment... Nov 24 05:12:53.903444022 (357h8m52.454914523s) [seg_pos={0 0}, upload=314373]: Done begin upload segment. Nov 24 05:12:53.903485471 (357h8m52.454955932s) [noise=true , address=45.91.116.6:4973 , piece_id=YFXRU76I3BSIAI3DKFRFERTVMJLR2CENQHYP3TMUCDBKBLWGGRYA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.903511801 (357h8m52.454982262s) [noise=true , address=77.90.31.14:4661 , piece_id=KLSYXSHBESAOQKDM7UIEJRBEU47LAZ4QLXQ7IIRHMGRHQDOTTGSQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.904736563 (357h8m52.456207024s) [noise=true , address=145.79.227.23:4547 , piece_id=77DA6ZG5K7FIM5VVFGLLIU3ZFPFSE6LYXA5ICFT32TGPTE4E6EWA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.905132177 (357h8m52.456602628s) [noise=true , address=145.79.227.6:4257 , piece_id=4YPKUHVNKCB25N5VNGRUAHIEAWOEIZL6FIM66XZGAI4SGNT6YV5Q, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.905529643 (357h8m52.457000104s) [noise=true , address=145.79.227.25:4619 , piece_id=Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906023413 (357h8m52.457493874s) [noise=true , address=145.79.227.22:4511 , piece_id=TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906454855 (357h8m52.457925306s) [noise=true , address=145.79.227.21:4474 , piece_id=3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.906891877 (357h8m52.458362338s) [noise=true , address=145.79.227.10:4402 , piece_id=JT3O5M6VTUIFKBZKLTCBHTQWCOEYNWF4G2VW6DR73MVPIEAERLBQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.908246807 (357h8m52.459717268s) [noise=true , address=145.79.227.15:4875 , piece_id=VUFSGD36YPWJ32JILLC3ATEW2NYLIA4H3ROPTEFYXQQNXYPUC6VA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.908390811 (357h8m52.459861272s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.909633959 (357h8m52.46110442s) [noise=true , address=145.79.227.24:4582 , piece_id=EXSXMMBRJ4AGORCP4YYSET432YNOTBYOU32O42BZOO42LYOGBJJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.910221468 (357h8m52.461691929s) [noise=true , address=77.90.31.13:4628 , piece_id=TO7ZMNG64OQLA4AFEZKTSBPPB7DGLE72Q2DHZRWMGN54X2FQ47TA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.911006863 (357h8m52.462477315s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.911862011 (357h8m52.463332462s) [noise=true , address=45.91.116.4:4937 , piece_id=SL3XPUODGLVL5SNMZU7A6BT57IQ7GVGZE343ZJVNSFWC5TX3GVVQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.912556263 (357h8m52.464026714s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:53.913294699 (357h8m52.46476516s) [noise=true , address=45.91.116.11:4118 , piece_id=NEVUNIGHGP5AQPONBJ5J3IJCVJTUYGKHQNITKVG2ZABNTUWWZIBA, seg_pos={0 0}, upload=314373]: Uploading piece... Nov 24 05:12:55.574621520 (357h8m54.126091981s) [upload=314373]: Next returned nil segment Nov 24 05:12:55.574639704 (357h8m54.126110165s) [upload=314373]: Waiting for error group managing segments... Nov 24 05:12:55.966933719 (357h8m54.51840418s) [noise=true , address=77.90.31.14:4661 , piece_id=KLSYXSHBESAOQKDM7UIEJRBEU47LAZ4QLXQ7IIRHMGRHQDOTTGSQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:55.981186689 (357h8m54.53265714s) [noise=true , address=45.91.116.11:4118 , piece_id=NEVUNIGHGP5AQPONBJ5J3IJCVJTUYGKHQNITKVG2ZABNTUWWZIBA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.045206764 (357h8m54.596677225s) [noise=true , address=45.91.116.6:4973 , piece_id=YFXRU76I3BSIAI3DKFRFERTVMJLR2CENQHYP3TMUCDBKBLWGGRYA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.047372809 (357h8m54.59884327s) [noise=true , address=77.90.31.13:4628 , piece_id=TO7ZMNG64OQLA4AFEZKTSBPPB7DGLE72Q2DHZRWMGN54X2FQ47TA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.056652799 (357h8m54.60812326s) [noise=true , address=145.79.227.6:4257 , piece_id=4YPKUHVNKCB25N5VNGRUAHIEAWOEIZL6FIM66XZGAI4SGNT6YV5Q, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.059483058 (357h8m54.610953519s) [noise=true , address=145.79.227.21:4474 , piece_id=3RDKLA34IDAPPB2HPZXFQNDXAUZRX4AE446QD3BB3FL4NBIJ5EHA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.070599767 (357h8m54.622070228s) [noise=true , address=145.79.227.10:4402 , piece_id=JT3O5M6VTUIFKBZKLTCBHTQWCOEYNWF4G2VW6DR73MVPIEAERLBQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.074296216 (357h8m54.625766677s) [noise=true , address=45.91.116.4:4937 , piece_id=SL3XPUODGLVL5SNMZU7A6BT57IQ7GVGZE343ZJVNSFWC5TX3GVVQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.081602177 (357h8m54.633072639s) [noise=true , address=145.79.227.22:4511 , piece_id=TDWB7E5APZHBVVXPRPJVSEX4LDNETDDGYCD7YH4SFAUQWFCUDUFA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.082195698 (357h8m54.633666149s) [noise=true , address=145.79.227.23:4547 , piece_id=77DA6ZG5K7FIM5VVFGLLIU3ZFPFSE6LYXA5ICFT32TGPTE4E6EWA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.089421227 (357h8m54.640891688s) [noise=true , address=145.79.227.24:4582 , piece_id=EXSXMMBRJ4AGORCP4YYSET432YNOTBYOU32O42BZOO42LYOGBJJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.094493995 (357h8m54.645964446s) [noise=true , address=145.79.227.15:4875 , piece_id=VUFSGD36YPWJ32JILLC3ATEW2NYLIA4H3ROPTEFYXQQNXYPUC6VA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.111919777 (357h8m54.663390248s) [noise=true , address=145.79.227.25:4619 , piece_id=Y47ELHUFN2X2AJVSAIHK4O65Y4KPAHNSMZEGSETXHKCYZYPKXDXQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: <nil> Nov 24 05:12:56.111947239 (357h8m54.6634177s) [seg_pos={0 0}, upload=314373]: Segment reached optimal threshold of 13 pieces. Nov 24 05:12:56.112360226 (357h8m54.663830707s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112371026 (357h8m54.663841487s) [noise=true , address=145.79.227.18:4006 , piece_id=PS7E6WQVLF6GJVNCYI2WRBUMBDCTRVVE3WX2TCJEGD766O553JBA, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112778422 (357h8m54.664248883s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112789903 (357h8m54.664260364s) [noise=true , address=77.90.31.10:4519 , piece_id=3RGI2HQHFGVVKGXT2SP75GW6CFUSMWEZUCMAOXJBXGNRW6LWWMJQ, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112805814 (357h8m54.664276265s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Nov 24 05:12:56.112813649 (357h8m54.66428411s) [noise=true , address=145.79.227.16:4910 , piece_id=PBZ4MWC4LWPBSK34S2SOUPEZTWB7LDV3OPAVZKBNXYXN5KGB4CJQ, seg_pos={0 0}, upload=314373]: Long tail cancellation due to optimal threshold Nov 24 05:12:56.112892599 (357h8m54.66436305s) [seg_pos={0 0}, upload=314373]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:12:56.113070117 (357h8m54.664540588s) [upload=314373]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:12:56.113079936 (357h8m54.664550387s) [upload=314373]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:12:56.609050819 (357h8m55.16052127s) [upload=314373]: Done uploading segments... Nov 24 05:12:56.609072030 (357h8m55.160542491s) [upload=314373]: Upload finished. err: <nil> Duration: 30.004561781s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.869114766s trace: https://jaeger.dev.storj.io/trace/62337c0a3cad45b3 FAIL (2025-11-24 05:12:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: TOYIVZ3Q3YYFTYM5QZ5JWZKOBVD5ZI66DZQMTS3SXC6JPRKET57Q) context deadline exceeded; piecestore: (Node ID: 1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, Piece ID: PYT5275W274I2AWBIDSBR6PWO2QNB47L5BVJU4FOOY2UVGKDQKPQ) context deadline exceeded ======LOGS====== .828008898s) [upload=314334]: Starting upload Nov 24 05:10:09.276620184 (357h6m7.828090635s) [upload=314334]: Done starting upload Nov 24 05:10:09.276805236 (357h6m7.828275707s) [upload=314334]: Uploading segments... Nov 24 05:10:09.276875560 (357h6m7.828346041s) [upload=314334]: Got next segment. Inline: false Nov 24 05:10:09.276917690 (357h6m7.828388151s) [upload=314334]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:10:09.276924031 (357h6m7.828394493s) [upload=314334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:10:09.537919732 (357h6m8.089390213s) [seg_pos={0 0}, upload=314334]: Begin upload segment... Nov 24 05:10:09.537982552 (357h6m8.089453023s) [seg_pos={0 0}, upload=314334]: Done begin upload segment. Nov 24 05:10:09.538026094 (357h6m8.089496555s) [noise=true , address=145.79.227.26:4655 , piece_id=45RGIKXPWVRDFW722APAVG25ZPWF3VNCMP43SSGPG3EJ5XMYTUUQ, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.539061125 (357h6m8.090531586s) [noise=true , address=145.79.227.17:4969 , piece_id=LUBTPEWJNEZGMDP6DNHA5WBUO7SXPUSAYZCX5SBAD6S3ES4O5J7A, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.539529017 (357h6m8.090999478s) [noise=true , address=145.79.227.30:4763 , piece_id=L3QZ47NAYQULRYSYGDPNABOULRYVL5YG3I53TCPUDZRPRQAZJJMA, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.540037304 (357h6m8.091507765s) [noise=true , address=77.90.31.17:4770 , piece_id=ZNOYAS7FAQ4KY6N7TG7GNZWTMFWKTNBMNKYDVAT65X374WNYRA2A, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.540504755 (357h6m8.091975216s) [noise=true , address=145.79.227.9:4367 , piece_id=QQV57EY4FPEKDKHINFGWR3UMBDEQLFDAWTD7XEVWCLANKITA4J3Q, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.540930626 (357h6m8.092401087s) [noise=true , address=77.90.31.13:4628 , piece_id=QK5GLVS4KXOSVETUMGPWFFDEQDXTUEJOOQQ3KD47YQTPT2VLO35A, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.541352429 (357h6m8.09282289s) [noise=true , address=45.91.116.6:4973 , piece_id=XNXVL7RYBIGF66F57UW3RD3EVM62U4JTZOX2HGEGXDPDNWSTFP7Q, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.543356816 (357h6m8.094827277s) [noise=true , address=145.79.227.4:4186 , piece_id=CS65BNCSI4CRN6OX6CHS5R7TNKNWJ2NLMDVOMNGONBENPHUHH5WA, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.545503583 (357h6m8.096974044s) [noise=true , address=145.79.227.30:4762 , piece_id=DMES3JUU5S4FTK3ZMLGGP4KAD233G7OWW3M3VDPQUIVTULUSLWNA, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.547607669 (357h6m8.09907814s) [noise=true , address=145.79.227.31:4798 , piece_id=VYQHE7TQU4UEZN2C4TQUVXCO6D6U35DEMZR3V2OMENRZDS54OODA, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.549257702 (357h6m8.100728183s) [noise=true , address=45.91.116.4:4937 , piece_id=TL2BU6MZSY7LGCUULTA762NQNZI4O2WJZIYPRHCNS3GMTBIBXHDQ, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.551326170 (357h6m8.102796631s) [noise=true , address=145.79.227.22:4510 , piece_id=5HM3NG6UTLYXE6I3SAI7LWFZ5NNO4RBURKABPXFL6LSD5KXDZHZQ, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.553650676 (357h6m8.105121137s) [noise=true , address=145.79.227.8:4331 , piece_id=PYT5275W274I2AWBIDSBR6PWO2QNB47L5BVJU4FOOY2UVGKDQKPQ, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.555147637 (357h6m8.106618098s) [noise=true , address=145.79.227.25:4619 , piece_id=TOYIVZ3Q3YYFTYM5QZ5JWZKOBVD5ZI66DZQMTS3SXC6JPRKET57Q, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.556138142 (357h6m8.107608614s) [noise=true , address=145.79.227.15:4874 , piece_id=MJESEFDCBP5M46YRDHA2APXHEDQ2UTYAHC6I575B3OJ3U6VLB4UA, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.557493874 (357h6m8.108964345s) [noise=true , address=77.90.31.11:4555 , piece_id=FTKZ6C274BRKOCWHDKGMQSI7HALI5EM3OFSV3JDJ3KF3MULEI33Q, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:11.331546183 (357h6m9.883016634s) [upload=314334]: Next returned nil segment Nov 24 05:10:11.331563105 (357h6m9.883033556s) [upload=314334]: Waiting for error group managing segments... Nov 24 05:10:11.629785241 (357h6m10.181255702s) [noise=true , address=45.91.116.4:4937 , piece_id=TL2BU6MZSY7LGCUULTA762NQNZI4O2WJZIYPRHCNS3GMTBIBXHDQ, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.676789805 (357h6m10.228260266s) [noise=true , address=77.90.31.13:4628 , piece_id=QK5GLVS4KXOSVETUMGPWFFDEQDXTUEJOOQQ3KD47YQTPT2VLO35A, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.686267282 (357h6m10.237737743s) [noise=true , address=145.79.227.17:4969 , piece_id=LUBTPEWJNEZGMDP6DNHA5WBUO7SXPUSAYZCX5SBAD6S3ES4O5J7A, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.695469755 (357h6m10.246940206s) [noise=true , address=145.79.227.30:4762 , piece_id=DMES3JUU5S4FTK3ZMLGGP4KAD233G7OWW3M3VDPQUIVTULUSLWNA, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.696669821 (357h6m10.248140282s) [noise=true , address=45.91.116.6:4973 , piece_id=XNXVL7RYBIGF66F57UW3RD3EVM62U4JTZOX2HGEGXDPDNWSTFP7Q, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.698880219 (357h6m10.25035068s) [noise=true , address=145.79.227.22:4510 , piece_id=5HM3NG6UTLYXE6I3SAI7LWFZ5NNO4RBURKABPXFL6LSD5KXDZHZQ, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.699635347 (357h6m10.251105819s) [noise=true , address=145.79.227.30:4763 , piece_id=L3QZ47NAYQULRYSYGDPNABOULRYVL5YG3I53TCPUDZRPRQAZJJMA, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.718815840 (357h6m10.270286301s) [noise=true , address=145.79.227.25:4619 , piece_id=TOYIVZ3Q3YYFTYM5QZ5JWZKOBVD5ZI66DZQMTS3SXC6JPRKET57Q, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.723472568 (357h6m10.274943029s) [noise=true , address=145.79.227.8:4331 , piece_id=PYT5275W274I2AWBIDSBR6PWO2QNB47L5BVJU4FOOY2UVGKDQKPQ, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.750701967 (357h6m10.302172458s) [noise=true , address=145.79.227.26:4655 , piece_id=45RGIKXPWVRDFW722APAVG25ZPWF3VNCMP43SSGPG3EJ5XMYTUUQ, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.769754877 (357h6m10.321225328s) [noise=true , address=77.90.31.11:4555 , piece_id=FTKZ6C274BRKOCWHDKGMQSI7HALI5EM3OFSV3JDJ3KF3MULEI33Q, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.812473200 (357h6m10.363943661s) [noise=true , address=145.79.227.31:4798 , piece_id=VYQHE7TQU4UEZN2C4TQUVXCO6D6U35DEMZR3V2OMENRZDS54OODA, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.825618331 (357h6m10.377088792s) [noise=true , address=77.90.31.17:4770 , piece_id=ZNOYAS7FAQ4KY6N7TG7GNZWTMFWKTNBMNKYDVAT65X374WNYRA2A, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.825643217 (357h6m10.377113678s) [seg_pos={0 0}, upload=314334]: Segment reached optimal threshold of 13 pieces. Nov 24 05:10:11.825978216 (357h6m10.377448687s) [noise=true , address=145.79.227.9:4367 , piece_id=QQV57EY4FPEKDKHINFGWR3UMBDEQLFDAWTD7XEVWCLANKITA4J3Q, seg_pos={0 0}, upload=314334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: read tcp 188.166.225.170:42150->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 24 05:10:11.825986311 (357h6m10.377456782s) [noise=true , address=145.79.227.9:4367 , piece_id=QQV57EY4FPEKDKHINFGWR3UMBDEQLFDAWTD7XEVWCLANKITA4J3Q, seg_pos={0 0}, upload=314334]: Long tail cancellation due to optimal threshold Nov 24 05:10:11.826020807 (357h6m10.377491268s) [noise=true , address=145.79.227.4:4186 , piece_id=CS65BNCSI4CRN6OX6CHS5R7TNKNWJ2NLMDVOMNGONBENPHUHH5WA, seg_pos={0 0}, upload=314334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: write tcp 188.166.225.170:56832->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56832->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56832->145.79.227.4:4186: use of closed network connection Nov 24 05:10:11.826031737 (357h6m10.377502198s) [noise=true , address=145.79.227.4:4186 , piece_id=CS65BNCSI4CRN6OX6CHS5R7TNKNWJ2NLMDVOMNGONBENPHUHH5WA, seg_pos={0 0}, upload=314334]: Long tail cancellation due to optimal threshold Nov 24 05:10:11.826243952 (357h6m10.377714423s) [noise=true , address=145.79.227.15:4874 , piece_id=MJESEFDCBP5M46YRDHA2APXHEDQ2UTYAHC6I575B3OJ3U6VLB4UA, seg_pos={0 0}, upload=314334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Nov 24 05:10:11.826252879 (357h6m10.37772334s) [noise=true , address=145.79.227.15:4874 , piece_id=MJESEFDCBP5M46YRDHA2APXHEDQ2UTYAHC6I575B3OJ3U6VLB4UA, seg_pos={0 0}, upload=314334]: Long tail cancellation due to optimal threshold Nov 24 05:10:11.826282806 (357h6m10.377753257s) [seg_pos={0 0}, upload=314334]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:10:11.826394599 (357h6m10.37786506s) [upload=314334]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:10:11.826402674 (357h6m10.377873125s) [upload=314334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:10:12.448269894 (357h6m10.999740355s) [upload=314334]: Done uploading segments... Nov 24 05:10:12.448291876 (357h6m10.999762337s) [upload=314334]: Upload finished. err: <nil> Duration: 5.86914304s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.84515878s trace: https://jaeger.dev.storj.io/trace/62337c0a3cad45b3 FAIL (2025-11-24 05:11:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: TOYIVZ3Q3YYFTYM5QZ5JWZKOBVD5ZI66DZQMTS3SXC6JPRKET57Q) context deadline exceeded; piecestore: (Node ID: 1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, Piece ID: PYT5275W274I2AWBIDSBR6PWO2QNB47L5BVJU4FOOY2UVGKDQKPQ) context deadline exceeded ======LOGS====== .828008898s) [upload=314334]: Starting upload Nov 24 05:10:09.276620184 (357h6m7.828090635s) [upload=314334]: Done starting upload Nov 24 05:10:09.276805236 (357h6m7.828275707s) [upload=314334]: Uploading segments... Nov 24 05:10:09.276875560 (357h6m7.828346041s) [upload=314334]: Got next segment. Inline: false Nov 24 05:10:09.276917690 (357h6m7.828388151s) [upload=314334]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:10:09.276924031 (357h6m7.828394493s) [upload=314334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:10:09.537919732 (357h6m8.089390213s) [seg_pos={0 0}, upload=314334]: Begin upload segment... Nov 24 05:10:09.537982552 (357h6m8.089453023s) [seg_pos={0 0}, upload=314334]: Done begin upload segment. Nov 24 05:10:09.538026094 (357h6m8.089496555s) [noise=true , address=145.79.227.26:4655 , piece_id=45RGIKXPWVRDFW722APAVG25ZPWF3VNCMP43SSGPG3EJ5XMYTUUQ, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.539061125 (357h6m8.090531586s) [noise=true , address=145.79.227.17:4969 , piece_id=LUBTPEWJNEZGMDP6DNHA5WBUO7SXPUSAYZCX5SBAD6S3ES4O5J7A, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.539529017 (357h6m8.090999478s) [noise=true , address=145.79.227.30:4763 , piece_id=L3QZ47NAYQULRYSYGDPNABOULRYVL5YG3I53TCPUDZRPRQAZJJMA, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.540037304 (357h6m8.091507765s) [noise=true , address=77.90.31.17:4770 , piece_id=ZNOYAS7FAQ4KY6N7TG7GNZWTMFWKTNBMNKYDVAT65X374WNYRA2A, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.540504755 (357h6m8.091975216s) [noise=true , address=145.79.227.9:4367 , piece_id=QQV57EY4FPEKDKHINFGWR3UMBDEQLFDAWTD7XEVWCLANKITA4J3Q, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.540930626 (357h6m8.092401087s) [noise=true , address=77.90.31.13:4628 , piece_id=QK5GLVS4KXOSVETUMGPWFFDEQDXTUEJOOQQ3KD47YQTPT2VLO35A, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.541352429 (357h6m8.09282289s) [noise=true , address=45.91.116.6:4973 , piece_id=XNXVL7RYBIGF66F57UW3RD3EVM62U4JTZOX2HGEGXDPDNWSTFP7Q, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.543356816 (357h6m8.094827277s) [noise=true , address=145.79.227.4:4186 , piece_id=CS65BNCSI4CRN6OX6CHS5R7TNKNWJ2NLMDVOMNGONBENPHUHH5WA, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.545503583 (357h6m8.096974044s) [noise=true , address=145.79.227.30:4762 , piece_id=DMES3JUU5S4FTK3ZMLGGP4KAD233G7OWW3M3VDPQUIVTULUSLWNA, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.547607669 (357h6m8.09907814s) [noise=true , address=145.79.227.31:4798 , piece_id=VYQHE7TQU4UEZN2C4TQUVXCO6D6U35DEMZR3V2OMENRZDS54OODA, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.549257702 (357h6m8.100728183s) [noise=true , address=45.91.116.4:4937 , piece_id=TL2BU6MZSY7LGCUULTA762NQNZI4O2WJZIYPRHCNS3GMTBIBXHDQ, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.551326170 (357h6m8.102796631s) [noise=true , address=145.79.227.22:4510 , piece_id=5HM3NG6UTLYXE6I3SAI7LWFZ5NNO4RBURKABPXFL6LSD5KXDZHZQ, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.553650676 (357h6m8.105121137s) [noise=true , address=145.79.227.8:4331 , piece_id=PYT5275W274I2AWBIDSBR6PWO2QNB47L5BVJU4FOOY2UVGKDQKPQ, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.555147637 (357h6m8.106618098s) [noise=true , address=145.79.227.25:4619 , piece_id=TOYIVZ3Q3YYFTYM5QZ5JWZKOBVD5ZI66DZQMTS3SXC6JPRKET57Q, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.556138142 (357h6m8.107608614s) [noise=true , address=145.79.227.15:4874 , piece_id=MJESEFDCBP5M46YRDHA2APXHEDQ2UTYAHC6I575B3OJ3U6VLB4UA, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.557493874 (357h6m8.108964345s) [noise=true , address=77.90.31.11:4555 , piece_id=FTKZ6C274BRKOCWHDKGMQSI7HALI5EM3OFSV3JDJ3KF3MULEI33Q, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:11.331546183 (357h6m9.883016634s) [upload=314334]: Next returned nil segment Nov 24 05:10:11.331563105 (357h6m9.883033556s) [upload=314334]: Waiting for error group managing segments... Nov 24 05:10:11.629785241 (357h6m10.181255702s) [noise=true , address=45.91.116.4:4937 , piece_id=TL2BU6MZSY7LGCUULTA762NQNZI4O2WJZIYPRHCNS3GMTBIBXHDQ, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.676789805 (357h6m10.228260266s) [noise=true , address=77.90.31.13:4628 , piece_id=QK5GLVS4KXOSVETUMGPWFFDEQDXTUEJOOQQ3KD47YQTPT2VLO35A, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.686267282 (357h6m10.237737743s) [noise=true , address=145.79.227.17:4969 , piece_id=LUBTPEWJNEZGMDP6DNHA5WBUO7SXPUSAYZCX5SBAD6S3ES4O5J7A, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.695469755 (357h6m10.246940206s) [noise=true , address=145.79.227.30:4762 , piece_id=DMES3JUU5S4FTK3ZMLGGP4KAD233G7OWW3M3VDPQUIVTULUSLWNA, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.696669821 (357h6m10.248140282s) [noise=true , address=45.91.116.6:4973 , piece_id=XNXVL7RYBIGF66F57UW3RD3EVM62U4JTZOX2HGEGXDPDNWSTFP7Q, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.698880219 (357h6m10.25035068s) [noise=true , address=145.79.227.22:4510 , piece_id=5HM3NG6UTLYXE6I3SAI7LWFZ5NNO4RBURKABPXFL6LSD5KXDZHZQ, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.699635347 (357h6m10.251105819s) [noise=true , address=145.79.227.30:4763 , piece_id=L3QZ47NAYQULRYSYGDPNABOULRYVL5YG3I53TCPUDZRPRQAZJJMA, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.718815840 (357h6m10.270286301s) [noise=true , address=145.79.227.25:4619 , piece_id=TOYIVZ3Q3YYFTYM5QZ5JWZKOBVD5ZI66DZQMTS3SXC6JPRKET57Q, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.723472568 (357h6m10.274943029s) [noise=true , address=145.79.227.8:4331 , piece_id=PYT5275W274I2AWBIDSBR6PWO2QNB47L5BVJU4FOOY2UVGKDQKPQ, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.750701967 (357h6m10.302172458s) [noise=true , address=145.79.227.26:4655 , piece_id=45RGIKXPWVRDFW722APAVG25ZPWF3VNCMP43SSGPG3EJ5XMYTUUQ, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.769754877 (357h6m10.321225328s) [noise=true , address=77.90.31.11:4555 , piece_id=FTKZ6C274BRKOCWHDKGMQSI7HALI5EM3OFSV3JDJ3KF3MULEI33Q, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.812473200 (357h6m10.363943661s) [noise=true , address=145.79.227.31:4798 , piece_id=VYQHE7TQU4UEZN2C4TQUVXCO6D6U35DEMZR3V2OMENRZDS54OODA, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.825618331 (357h6m10.377088792s) [noise=true , address=77.90.31.17:4770 , piece_id=ZNOYAS7FAQ4KY6N7TG7GNZWTMFWKTNBMNKYDVAT65X374WNYRA2A, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.825643217 (357h6m10.377113678s) [seg_pos={0 0}, upload=314334]: Segment reached optimal threshold of 13 pieces. Nov 24 05:10:11.825978216 (357h6m10.377448687s) [noise=true , address=145.79.227.9:4367 , piece_id=QQV57EY4FPEKDKHINFGWR3UMBDEQLFDAWTD7XEVWCLANKITA4J3Q, seg_pos={0 0}, upload=314334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: read tcp 188.166.225.170:42150->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 24 05:10:11.825986311 (357h6m10.377456782s) [noise=true , address=145.79.227.9:4367 , piece_id=QQV57EY4FPEKDKHINFGWR3UMBDEQLFDAWTD7XEVWCLANKITA4J3Q, seg_pos={0 0}, upload=314334]: Long tail cancellation due to optimal threshold Nov 24 05:10:11.826020807 (357h6m10.377491268s) [noise=true , address=145.79.227.4:4186 , piece_id=CS65BNCSI4CRN6OX6CHS5R7TNKNWJ2NLMDVOMNGONBENPHUHH5WA, seg_pos={0 0}, upload=314334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: write tcp 188.166.225.170:56832->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56832->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56832->145.79.227.4:4186: use of closed network connection Nov 24 05:10:11.826031737 (357h6m10.377502198s) [noise=true , address=145.79.227.4:4186 , piece_id=CS65BNCSI4CRN6OX6CHS5R7TNKNWJ2NLMDVOMNGONBENPHUHH5WA, seg_pos={0 0}, upload=314334]: Long tail cancellation due to optimal threshold Nov 24 05:10:11.826243952 (357h6m10.377714423s) [noise=true , address=145.79.227.15:4874 , piece_id=MJESEFDCBP5M46YRDHA2APXHEDQ2UTYAHC6I575B3OJ3U6VLB4UA, seg_pos={0 0}, upload=314334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Nov 24 05:10:11.826252879 (357h6m10.37772334s) [noise=true , address=145.79.227.15:4874 , piece_id=MJESEFDCBP5M46YRDHA2APXHEDQ2UTYAHC6I575B3OJ3U6VLB4UA, seg_pos={0 0}, upload=314334]: Long tail cancellation due to optimal threshold Nov 24 05:10:11.826282806 (357h6m10.377753257s) [seg_pos={0 0}, upload=314334]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:10:11.826394599 (357h6m10.37786506s) [upload=314334]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:10:11.826402674 (357h6m10.377873125s) [upload=314334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:10:12.448269894 (357h6m10.999740355s) [upload=314334]: Done uploading segments... Nov 24 05:10:12.448291876 (357h6m10.999762337s) [upload=314334]: Upload finished. err: <nil> Duration: 5.845190119s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.714143474s trace: https://jaeger.dev.storj.io/trace/62337c0a3cad45b3 FAIL (2025-11-24 05:10:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: TOYIVZ3Q3YYFTYM5QZ5JWZKOBVD5ZI66DZQMTS3SXC6JPRKET57Q) context deadline exceeded; piecestore: (Node ID: 1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, Piece ID: PYT5275W274I2AWBIDSBR6PWO2QNB47L5BVJU4FOOY2UVGKDQKPQ) context deadline exceeded ======LOGS====== .828008898s) [upload=314334]: Starting upload Nov 24 05:10:09.276620184 (357h6m7.828090635s) [upload=314334]: Done starting upload Nov 24 05:10:09.276805236 (357h6m7.828275707s) [upload=314334]: Uploading segments... Nov 24 05:10:09.276875560 (357h6m7.828346041s) [upload=314334]: Got next segment. Inline: false Nov 24 05:10:09.276917690 (357h6m7.828388151s) [upload=314334]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:10:09.276924031 (357h6m7.828394493s) [upload=314334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:10:09.537919732 (357h6m8.089390213s) [seg_pos={0 0}, upload=314334]: Begin upload segment... Nov 24 05:10:09.537982552 (357h6m8.089453023s) [seg_pos={0 0}, upload=314334]: Done begin upload segment. Nov 24 05:10:09.538026094 (357h6m8.089496555s) [noise=true , address=145.79.227.26:4655 , piece_id=45RGIKXPWVRDFW722APAVG25ZPWF3VNCMP43SSGPG3EJ5XMYTUUQ, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.539061125 (357h6m8.090531586s) [noise=true , address=145.79.227.17:4969 , piece_id=LUBTPEWJNEZGMDP6DNHA5WBUO7SXPUSAYZCX5SBAD6S3ES4O5J7A, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.539529017 (357h6m8.090999478s) [noise=true , address=145.79.227.30:4763 , piece_id=L3QZ47NAYQULRYSYGDPNABOULRYVL5YG3I53TCPUDZRPRQAZJJMA, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.540037304 (357h6m8.091507765s) [noise=true , address=77.90.31.17:4770 , piece_id=ZNOYAS7FAQ4KY6N7TG7GNZWTMFWKTNBMNKYDVAT65X374WNYRA2A, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.540504755 (357h6m8.091975216s) [noise=true , address=145.79.227.9:4367 , piece_id=QQV57EY4FPEKDKHINFGWR3UMBDEQLFDAWTD7XEVWCLANKITA4J3Q, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.540930626 (357h6m8.092401087s) [noise=true , address=77.90.31.13:4628 , piece_id=QK5GLVS4KXOSVETUMGPWFFDEQDXTUEJOOQQ3KD47YQTPT2VLO35A, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.541352429 (357h6m8.09282289s) [noise=true , address=45.91.116.6:4973 , piece_id=XNXVL7RYBIGF66F57UW3RD3EVM62U4JTZOX2HGEGXDPDNWSTFP7Q, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.543356816 (357h6m8.094827277s) [noise=true , address=145.79.227.4:4186 , piece_id=CS65BNCSI4CRN6OX6CHS5R7TNKNWJ2NLMDVOMNGONBENPHUHH5WA, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.545503583 (357h6m8.096974044s) [noise=true , address=145.79.227.30:4762 , piece_id=DMES3JUU5S4FTK3ZMLGGP4KAD233G7OWW3M3VDPQUIVTULUSLWNA, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.547607669 (357h6m8.09907814s) [noise=true , address=145.79.227.31:4798 , piece_id=VYQHE7TQU4UEZN2C4TQUVXCO6D6U35DEMZR3V2OMENRZDS54OODA, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.549257702 (357h6m8.100728183s) [noise=true , address=45.91.116.4:4937 , piece_id=TL2BU6MZSY7LGCUULTA762NQNZI4O2WJZIYPRHCNS3GMTBIBXHDQ, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.551326170 (357h6m8.102796631s) [noise=true , address=145.79.227.22:4510 , piece_id=5HM3NG6UTLYXE6I3SAI7LWFZ5NNO4RBURKABPXFL6LSD5KXDZHZQ, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.553650676 (357h6m8.105121137s) [noise=true , address=145.79.227.8:4331 , piece_id=PYT5275W274I2AWBIDSBR6PWO2QNB47L5BVJU4FOOY2UVGKDQKPQ, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.555147637 (357h6m8.106618098s) [noise=true , address=145.79.227.25:4619 , piece_id=TOYIVZ3Q3YYFTYM5QZ5JWZKOBVD5ZI66DZQMTS3SXC6JPRKET57Q, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.556138142 (357h6m8.107608614s) [noise=true , address=145.79.227.15:4874 , piece_id=MJESEFDCBP5M46YRDHA2APXHEDQ2UTYAHC6I575B3OJ3U6VLB4UA, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.557493874 (357h6m8.108964345s) [noise=true , address=77.90.31.11:4555 , piece_id=FTKZ6C274BRKOCWHDKGMQSI7HALI5EM3OFSV3JDJ3KF3MULEI33Q, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:11.331546183 (357h6m9.883016634s) [upload=314334]: Next returned nil segment Nov 24 05:10:11.331563105 (357h6m9.883033556s) [upload=314334]: Waiting for error group managing segments... Nov 24 05:10:11.629785241 (357h6m10.181255702s) [noise=true , address=45.91.116.4:4937 , piece_id=TL2BU6MZSY7LGCUULTA762NQNZI4O2WJZIYPRHCNS3GMTBIBXHDQ, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.676789805 (357h6m10.228260266s) [noise=true , address=77.90.31.13:4628 , piece_id=QK5GLVS4KXOSVETUMGPWFFDEQDXTUEJOOQQ3KD47YQTPT2VLO35A, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.686267282 (357h6m10.237737743s) [noise=true , address=145.79.227.17:4969 , piece_id=LUBTPEWJNEZGMDP6DNHA5WBUO7SXPUSAYZCX5SBAD6S3ES4O5J7A, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.695469755 (357h6m10.246940206s) [noise=true , address=145.79.227.30:4762 , piece_id=DMES3JUU5S4FTK3ZMLGGP4KAD233G7OWW3M3VDPQUIVTULUSLWNA, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.696669821 (357h6m10.248140282s) [noise=true , address=45.91.116.6:4973 , piece_id=XNXVL7RYBIGF66F57UW3RD3EVM62U4JTZOX2HGEGXDPDNWSTFP7Q, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.698880219 (357h6m10.25035068s) [noise=true , address=145.79.227.22:4510 , piece_id=5HM3NG6UTLYXE6I3SAI7LWFZ5NNO4RBURKABPXFL6LSD5KXDZHZQ, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.699635347 (357h6m10.251105819s) [noise=true , address=145.79.227.30:4763 , piece_id=L3QZ47NAYQULRYSYGDPNABOULRYVL5YG3I53TCPUDZRPRQAZJJMA, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.718815840 (357h6m10.270286301s) [noise=true , address=145.79.227.25:4619 , piece_id=TOYIVZ3Q3YYFTYM5QZ5JWZKOBVD5ZI66DZQMTS3SXC6JPRKET57Q, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.723472568 (357h6m10.274943029s) [noise=true , address=145.79.227.8:4331 , piece_id=PYT5275W274I2AWBIDSBR6PWO2QNB47L5BVJU4FOOY2UVGKDQKPQ, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.750701967 (357h6m10.302172458s) [noise=true , address=145.79.227.26:4655 , piece_id=45RGIKXPWVRDFW722APAVG25ZPWF3VNCMP43SSGPG3EJ5XMYTUUQ, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.769754877 (357h6m10.321225328s) [noise=true , address=77.90.31.11:4555 , piece_id=FTKZ6C274BRKOCWHDKGMQSI7HALI5EM3OFSV3JDJ3KF3MULEI33Q, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.812473200 (357h6m10.363943661s) [noise=true , address=145.79.227.31:4798 , piece_id=VYQHE7TQU4UEZN2C4TQUVXCO6D6U35DEMZR3V2OMENRZDS54OODA, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.825618331 (357h6m10.377088792s) [noise=true , address=77.90.31.17:4770 , piece_id=ZNOYAS7FAQ4KY6N7TG7GNZWTMFWKTNBMNKYDVAT65X374WNYRA2A, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.825643217 (357h6m10.377113678s) [seg_pos={0 0}, upload=314334]: Segment reached optimal threshold of 13 pieces. Nov 24 05:10:11.825978216 (357h6m10.377448687s) [noise=true , address=145.79.227.9:4367 , piece_id=QQV57EY4FPEKDKHINFGWR3UMBDEQLFDAWTD7XEVWCLANKITA4J3Q, seg_pos={0 0}, upload=314334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: read tcp 188.166.225.170:42150->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 24 05:10:11.825986311 (357h6m10.377456782s) [noise=true , address=145.79.227.9:4367 , piece_id=QQV57EY4FPEKDKHINFGWR3UMBDEQLFDAWTD7XEVWCLANKITA4J3Q, seg_pos={0 0}, upload=314334]: Long tail cancellation due to optimal threshold Nov 24 05:10:11.826020807 (357h6m10.377491268s) [noise=true , address=145.79.227.4:4186 , piece_id=CS65BNCSI4CRN6OX6CHS5R7TNKNWJ2NLMDVOMNGONBENPHUHH5WA, seg_pos={0 0}, upload=314334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: write tcp 188.166.225.170:56832->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56832->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56832->145.79.227.4:4186: use of closed network connection Nov 24 05:10:11.826031737 (357h6m10.377502198s) [noise=true , address=145.79.227.4:4186 , piece_id=CS65BNCSI4CRN6OX6CHS5R7TNKNWJ2NLMDVOMNGONBENPHUHH5WA, seg_pos={0 0}, upload=314334]: Long tail cancellation due to optimal threshold Nov 24 05:10:11.826243952 (357h6m10.377714423s) [noise=true , address=145.79.227.15:4874 , piece_id=MJESEFDCBP5M46YRDHA2APXHEDQ2UTYAHC6I575B3OJ3U6VLB4UA, seg_pos={0 0}, upload=314334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Nov 24 05:10:11.826252879 (357h6m10.37772334s) [noise=true , address=145.79.227.15:4874 , piece_id=MJESEFDCBP5M46YRDHA2APXHEDQ2UTYAHC6I575B3OJ3U6VLB4UA, seg_pos={0 0}, upload=314334]: Long tail cancellation due to optimal threshold Nov 24 05:10:11.826282806 (357h6m10.377753257s) [seg_pos={0 0}, upload=314334]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:10:11.826394599 (357h6m10.37786506s) [upload=314334]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:10:11.826402674 (357h6m10.377873125s) [upload=314334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:10:12.448269894 (357h6m10.999740355s) [upload=314334]: Done uploading segments... Nov 24 05:10:12.448291876 (357h6m10.999762337s) [upload=314334]: Upload finished. err: <nil> Duration: 5.714175024s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.003615694s trace: https://jaeger.dev.storj.io/trace/62337c0a3cad45b3 FAIL (2025-11-24 05:10:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: TOYIVZ3Q3YYFTYM5QZ5JWZKOBVD5ZI66DZQMTS3SXC6JPRKET57Q) context deadline exceeded; piecestore: (Node ID: 1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, Piece ID: PYT5275W274I2AWBIDSBR6PWO2QNB47L5BVJU4FOOY2UVGKDQKPQ) context deadline exceeded ======LOGS====== .828008898s) [upload=314334]: Starting upload Nov 24 05:10:09.276620184 (357h6m7.828090635s) [upload=314334]: Done starting upload Nov 24 05:10:09.276805236 (357h6m7.828275707s) [upload=314334]: Uploading segments... Nov 24 05:10:09.276875560 (357h6m7.828346041s) [upload=314334]: Got next segment. Inline: false Nov 24 05:10:09.276917690 (357h6m7.828388151s) [upload=314334]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 24 05:10:09.276924031 (357h6m7.828394493s) [upload=314334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 24 05:10:09.537919732 (357h6m8.089390213s) [seg_pos={0 0}, upload=314334]: Begin upload segment... Nov 24 05:10:09.537982552 (357h6m8.089453023s) [seg_pos={0 0}, upload=314334]: Done begin upload segment. Nov 24 05:10:09.538026094 (357h6m8.089496555s) [noise=true , address=145.79.227.26:4655 , piece_id=45RGIKXPWVRDFW722APAVG25ZPWF3VNCMP43SSGPG3EJ5XMYTUUQ, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.539061125 (357h6m8.090531586s) [noise=true , address=145.79.227.17:4969 , piece_id=LUBTPEWJNEZGMDP6DNHA5WBUO7SXPUSAYZCX5SBAD6S3ES4O5J7A, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.539529017 (357h6m8.090999478s) [noise=true , address=145.79.227.30:4763 , piece_id=L3QZ47NAYQULRYSYGDPNABOULRYVL5YG3I53TCPUDZRPRQAZJJMA, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.540037304 (357h6m8.091507765s) [noise=true , address=77.90.31.17:4770 , piece_id=ZNOYAS7FAQ4KY6N7TG7GNZWTMFWKTNBMNKYDVAT65X374WNYRA2A, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.540504755 (357h6m8.091975216s) [noise=true , address=145.79.227.9:4367 , piece_id=QQV57EY4FPEKDKHINFGWR3UMBDEQLFDAWTD7XEVWCLANKITA4J3Q, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.540930626 (357h6m8.092401087s) [noise=true , address=77.90.31.13:4628 , piece_id=QK5GLVS4KXOSVETUMGPWFFDEQDXTUEJOOQQ3KD47YQTPT2VLO35A, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.541352429 (357h6m8.09282289s) [noise=true , address=45.91.116.6:4973 , piece_id=XNXVL7RYBIGF66F57UW3RD3EVM62U4JTZOX2HGEGXDPDNWSTFP7Q, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.543356816 (357h6m8.094827277s) [noise=true , address=145.79.227.4:4186 , piece_id=CS65BNCSI4CRN6OX6CHS5R7TNKNWJ2NLMDVOMNGONBENPHUHH5WA, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.545503583 (357h6m8.096974044s) [noise=true , address=145.79.227.30:4762 , piece_id=DMES3JUU5S4FTK3ZMLGGP4KAD233G7OWW3M3VDPQUIVTULUSLWNA, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.547607669 (357h6m8.09907814s) [noise=true , address=145.79.227.31:4798 , piece_id=VYQHE7TQU4UEZN2C4TQUVXCO6D6U35DEMZR3V2OMENRZDS54OODA, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.549257702 (357h6m8.100728183s) [noise=true , address=45.91.116.4:4937 , piece_id=TL2BU6MZSY7LGCUULTA762NQNZI4O2WJZIYPRHCNS3GMTBIBXHDQ, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.551326170 (357h6m8.102796631s) [noise=true , address=145.79.227.22:4510 , piece_id=5HM3NG6UTLYXE6I3SAI7LWFZ5NNO4RBURKABPXFL6LSD5KXDZHZQ, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.553650676 (357h6m8.105121137s) [noise=true , address=145.79.227.8:4331 , piece_id=PYT5275W274I2AWBIDSBR6PWO2QNB47L5BVJU4FOOY2UVGKDQKPQ, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.555147637 (357h6m8.106618098s) [noise=true , address=145.79.227.25:4619 , piece_id=TOYIVZ3Q3YYFTYM5QZ5JWZKOBVD5ZI66DZQMTS3SXC6JPRKET57Q, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.556138142 (357h6m8.107608614s) [noise=true , address=145.79.227.15:4874 , piece_id=MJESEFDCBP5M46YRDHA2APXHEDQ2UTYAHC6I575B3OJ3U6VLB4UA, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:09.557493874 (357h6m8.108964345s) [noise=true , address=77.90.31.11:4555 , piece_id=FTKZ6C274BRKOCWHDKGMQSI7HALI5EM3OFSV3JDJ3KF3MULEI33Q, seg_pos={0 0}, upload=314334]: Uploading piece... Nov 24 05:10:11.331546183 (357h6m9.883016634s) [upload=314334]: Next returned nil segment Nov 24 05:10:11.331563105 (357h6m9.883033556s) [upload=314334]: Waiting for error group managing segments... Nov 24 05:10:11.629785241 (357h6m10.181255702s) [noise=true , address=45.91.116.4:4937 , piece_id=TL2BU6MZSY7LGCUULTA762NQNZI4O2WJZIYPRHCNS3GMTBIBXHDQ, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.676789805 (357h6m10.228260266s) [noise=true , address=77.90.31.13:4628 , piece_id=QK5GLVS4KXOSVETUMGPWFFDEQDXTUEJOOQQ3KD47YQTPT2VLO35A, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.686267282 (357h6m10.237737743s) [noise=true , address=145.79.227.17:4969 , piece_id=LUBTPEWJNEZGMDP6DNHA5WBUO7SXPUSAYZCX5SBAD6S3ES4O5J7A, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.695469755 (357h6m10.246940206s) [noise=true , address=145.79.227.30:4762 , piece_id=DMES3JUU5S4FTK3ZMLGGP4KAD233G7OWW3M3VDPQUIVTULUSLWNA, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.696669821 (357h6m10.248140282s) [noise=true , address=45.91.116.6:4973 , piece_id=XNXVL7RYBIGF66F57UW3RD3EVM62U4JTZOX2HGEGXDPDNWSTFP7Q, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.698880219 (357h6m10.25035068s) [noise=true , address=145.79.227.22:4510 , piece_id=5HM3NG6UTLYXE6I3SAI7LWFZ5NNO4RBURKABPXFL6LSD5KXDZHZQ, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.699635347 (357h6m10.251105819s) [noise=true , address=145.79.227.30:4763 , piece_id=L3QZ47NAYQULRYSYGDPNABOULRYVL5YG3I53TCPUDZRPRQAZJJMA, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.718815840 (357h6m10.270286301s) [noise=true , address=145.79.227.25:4619 , piece_id=TOYIVZ3Q3YYFTYM5QZ5JWZKOBVD5ZI66DZQMTS3SXC6JPRKET57Q, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.723472568 (357h6m10.274943029s) [noise=true , address=145.79.227.8:4331 , piece_id=PYT5275W274I2AWBIDSBR6PWO2QNB47L5BVJU4FOOY2UVGKDQKPQ, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.750701967 (357h6m10.302172458s) [noise=true , address=145.79.227.26:4655 , piece_id=45RGIKXPWVRDFW722APAVG25ZPWF3VNCMP43SSGPG3EJ5XMYTUUQ, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.769754877 (357h6m10.321225328s) [noise=true , address=77.90.31.11:4555 , piece_id=FTKZ6C274BRKOCWHDKGMQSI7HALI5EM3OFSV3JDJ3KF3MULEI33Q, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.812473200 (357h6m10.363943661s) [noise=true , address=145.79.227.31:4798 , piece_id=VYQHE7TQU4UEZN2C4TQUVXCO6D6U35DEMZR3V2OMENRZDS54OODA, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.825618331 (357h6m10.377088792s) [noise=true , address=77.90.31.17:4770 , piece_id=ZNOYAS7FAQ4KY6N7TG7GNZWTMFWKTNBMNKYDVAT65X374WNYRA2A, seg_pos={0 0}, upload=314334]: Done uploading piece. err: <nil> Nov 24 05:10:11.825643217 (357h6m10.377113678s) [seg_pos={0 0}, upload=314334]: Segment reached optimal threshold of 13 pieces. Nov 24 05:10:11.825978216 (357h6m10.377448687s) [noise=true , address=145.79.227.9:4367 , piece_id=QQV57EY4FPEKDKHINFGWR3UMBDEQLFDAWTD7XEVWCLANKITA4J3Q, seg_pos={0 0}, upload=314334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: read tcp 188.166.225.170:42150->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 24 05:10:11.825986311 (357h6m10.377456782s) [noise=true , address=145.79.227.9:4367 , piece_id=QQV57EY4FPEKDKHINFGWR3UMBDEQLFDAWTD7XEVWCLANKITA4J3Q, seg_pos={0 0}, upload=314334]: Long tail cancellation due to optimal threshold Nov 24 05:10:11.826020807 (357h6m10.377491268s) [noise=true , address=145.79.227.4:4186 , piece_id=CS65BNCSI4CRN6OX6CHS5R7TNKNWJ2NLMDVOMNGONBENPHUHH5WA, seg_pos={0 0}, upload=314334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: write tcp 188.166.225.170:56832->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:56832->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56832->145.79.227.4:4186: use of closed network connection Nov 24 05:10:11.826031737 (357h6m10.377502198s) [noise=true , address=145.79.227.4:4186 , piece_id=CS65BNCSI4CRN6OX6CHS5R7TNKNWJ2NLMDVOMNGONBENPHUHH5WA, seg_pos={0 0}, upload=314334]: Long tail cancellation due to optimal threshold Nov 24 05:10:11.826243952 (357h6m10.377714423s) [noise=true , address=145.79.227.15:4874 , piece_id=MJESEFDCBP5M46YRDHA2APXHEDQ2UTYAHC6I575B3OJ3U6VLB4UA, seg_pos={0 0}, upload=314334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Nov 24 05:10:11.826252879 (357h6m10.37772334s) [noise=true , address=145.79.227.15:4874 , piece_id=MJESEFDCBP5M46YRDHA2APXHEDQ2UTYAHC6I575B3OJ3U6VLB4UA, seg_pos={0 0}, upload=314334]: Long tail cancellation due to optimal threshold Nov 24 05:10:11.826282806 (357h6m10.377753257s) [seg_pos={0 0}, upload=314334]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 24 05:10:11.826394599 (357h6m10.37786506s) [upload=314334]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 24 05:10:11.826402674 (357h6m10.377873125s) [upload=314334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 24 05:10:12.448269894 (357h6m10.999740355s) [upload=314334]: Done uploading segments... Nov 24 05:10:12.448291876 (357h6m10.999762337s) [upload=314334]: Upload finished. err: <nil> Duration: 30.003633007s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.424357651s trace: https://jaeger.dev.storj.io/trace/6aa8047b29749dfb FAIL (2025-11-24 00:57:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.424389181s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.454516235s trace: https://jaeger.dev.storj.io/trace/6aa8047b29749dfb FAIL (2025-11-24 00:56:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.454541525s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.537022851s trace: https://jaeger.dev.storj.io/trace/6aa8047b29749dfb FAIL (2025-11-24 00:55:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.537044351s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.466095868s trace: https://jaeger.dev.storj.io/trace/6aa8047b29749dfb FAIL (2025-11-24 00:54:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.46612355s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.410176971s trace: https://jaeger.dev.storj.io/trace/6aa8047b29749dfb FAIL (2025-11-24 00:54:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.410203782s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.544280887s trace: https://jaeger.dev.storj.io/trace/6aa8047b29749dfb FAIL (2025-11-24 00:54:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.544310522s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.47966148s trace: https://jaeger.dev.storj.io/trace/6aa8047b29749dfb FAIL (2025-11-24 00:53:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.479689874s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.510083308s trace: https://jaeger.dev.storj.io/trace/6aa8047b29749dfb FAIL (2025-11-24 00:52:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.510132692s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.395475145s trace: https://jaeger.dev.storj.io/trace/6aa8047b29749dfb FAIL (2025-11-24 00:51:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.395501506s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.460237756s trace: https://jaeger.dev.storj.io/trace/6aa8047b29749dfb FAIL (2025-11-24 00:50:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.460262383s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.459998399s trace: https://jaeger.dev.storj.io/trace/6aa8047b29749dfb FAIL (2025-11-24 00:50:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.460032814s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.46998368s trace: https://jaeger.dev.storj.io/trace/6aa8047b29749dfb FAIL (2025-11-24 00:48:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.470005531s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.490909428s trace: https://jaeger.dev.storj.io/trace/6aa8047b29749dfb FAIL (2025-11-24 00:48:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.490928053s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.556903981s trace: https://jaeger.dev.storj.io/trace/6aa8047b29749dfb FAIL (2025-11-24 00:47:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.556924821s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)