Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.807219768s trace: https://jaeger.dev.storj.io/trace/2dbb9d950879d488 FAIL (2025-12-07 17:18:30 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 17:04:40.750068351 (49h35m40.621151396s) [upload=43608]: Starting upload Dec 7 17:04:40.750151529 (49h35m40.621234574s) [upload=43608]: Done starting upload Dec 7 17:04:40.750252862 (49h35m40.621335927s) [upload=43608]: Uploading segments... Dec 7 17:04:40.750449186 (49h35m40.621532241s) [upload=43608]: Got next segment. Inline: false Dec 7 17:04:40.750495423 (49h35m40.621578459s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 17:04:40.750502757 (49h35m40.621585802s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 17:04:41.012862673 (49h35m40.883945718s) [seg_pos={0 0}, upload=43608]: Begin upload segment... Dec 7 17:04:41.012921534 (49h35m40.88400458s) [seg_pos={0 0}, upload=43608]: Done begin upload segment. Dec 7 17:04:41.012952173 (49h35m40.884035218s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.014639106 (49h35m40.885722151s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.015553246 (49h35m40.886636291s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016060653 (49h35m40.887143698s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016721020 (49h35m40.887804065s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.018348218 (49h35m40.889431273s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.020177682 (49h35m40.891260727s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.021303415 (49h35m40.89238646s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.023085799 (49h35m40.894168844s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.024447463 (49h35m40.895530498s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.025964030 (49h35m40.897047065s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.027289063 (49h35m40.898372108s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.028891215 (49h35m40.89997425s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.031185423 (49h35m40.902268468s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.032762747 (49h35m40.903845782s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.034246312 (49h35m40.905329347s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:42.710512985 (49h35m42.58159602s) [upload=43608]: Next returned nil segment Dec 7 17:04:42.710525298 (49h35m42.581608343s) [upload=43608]: Waiting for error group managing segments... Dec 7 17:04:43.179511201 (49h35m43.050594246s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:46.586841777 (49h35m46.457924822s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:53.671123584 (49h35m53.542206629s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:55.169666550 (49h35m55.040749595s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:57.066377404 (49h35m56.937460449s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:00.087003610 (49h35m59.958086645s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.688144215 (49h36m2.55922726s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.873597685 (49h36m2.74468073s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:03.747085322 (49h36m3.618168377s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.570204756 (49h36m4.441287801s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.766016562 (49h36m4.637099607s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:06.426653835 (49h36m6.29773689s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621680284 (49h36m7.492763339s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621710732 (49h36m7.492793777s) [seg_pos={0 0}, upload=43608]: Segment reached optimal threshold of 13 pieces. Dec 7 17:05:07.622110042 (49h36m7.493193097s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection Dec 7 17:05:07.622121895 (49h36m7.49320494s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622169114 (49h36m7.49325216s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: context canceled; context canceled Dec 7 17:05:07.622181257 (49h36m7.493264302s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622414712 (49h36m7.493497757s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: read tcp 188.166.225.170:41952->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 17:05:07.622423679 (49h36m7.493506724s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622449889 (49h36m7.493532924s) [seg_pos={0 0}, upload=43608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 17:05:07.622527457 (49h36m7.493610502s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 17:05:07.622532175 (49h36m7.49361522s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 17:05:07.933366757 (49h36m7.804449802s) [upload=43608]: Done uploading segments... Dec 7 17:05:07.933395712 (49h36m7.804478757s) [upload=43608]: Upload finished. err: <nil> Duration: 5.80724645s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.804236901s trace: https://jaeger.dev.storj.io/trace/2dbb9d950879d488 FAIL (2025-12-07 17:17:54 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 17:04:40.750068351 (49h35m40.621151396s) [upload=43608]: Starting upload Dec 7 17:04:40.750151529 (49h35m40.621234574s) [upload=43608]: Done starting upload Dec 7 17:04:40.750252862 (49h35m40.621335927s) [upload=43608]: Uploading segments... Dec 7 17:04:40.750449186 (49h35m40.621532241s) [upload=43608]: Got next segment. Inline: false Dec 7 17:04:40.750495423 (49h35m40.621578459s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 17:04:40.750502757 (49h35m40.621585802s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 17:04:41.012862673 (49h35m40.883945718s) [seg_pos={0 0}, upload=43608]: Begin upload segment... Dec 7 17:04:41.012921534 (49h35m40.88400458s) [seg_pos={0 0}, upload=43608]: Done begin upload segment. Dec 7 17:04:41.012952173 (49h35m40.884035218s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.014639106 (49h35m40.885722151s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.015553246 (49h35m40.886636291s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016060653 (49h35m40.887143698s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016721020 (49h35m40.887804065s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.018348218 (49h35m40.889431273s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.020177682 (49h35m40.891260727s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.021303415 (49h35m40.89238646s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.023085799 (49h35m40.894168844s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.024447463 (49h35m40.895530498s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.025964030 (49h35m40.897047065s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.027289063 (49h35m40.898372108s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.028891215 (49h35m40.89997425s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.031185423 (49h35m40.902268468s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.032762747 (49h35m40.903845782s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.034246312 (49h35m40.905329347s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:42.710512985 (49h35m42.58159602s) [upload=43608]: Next returned nil segment Dec 7 17:04:42.710525298 (49h35m42.581608343s) [upload=43608]: Waiting for error group managing segments... Dec 7 17:04:43.179511201 (49h35m43.050594246s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:46.586841777 (49h35m46.457924822s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:53.671123584 (49h35m53.542206629s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:55.169666550 (49h35m55.040749595s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:57.066377404 (49h35m56.937460449s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:00.087003610 (49h35m59.958086645s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.688144215 (49h36m2.55922726s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.873597685 (49h36m2.74468073s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:03.747085322 (49h36m3.618168377s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.570204756 (49h36m4.441287801s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.766016562 (49h36m4.637099607s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:06.426653835 (49h36m6.29773689s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621680284 (49h36m7.492763339s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621710732 (49h36m7.492793777s) [seg_pos={0 0}, upload=43608]: Segment reached optimal threshold of 13 pieces. Dec 7 17:05:07.622110042 (49h36m7.493193097s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection Dec 7 17:05:07.622121895 (49h36m7.49320494s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622169114 (49h36m7.49325216s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: context canceled; context canceled Dec 7 17:05:07.622181257 (49h36m7.493264302s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622414712 (49h36m7.493497757s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: read tcp 188.166.225.170:41952->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 17:05:07.622423679 (49h36m7.493506724s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622449889 (49h36m7.493532924s) [seg_pos={0 0}, upload=43608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 17:05:07.622527457 (49h36m7.493610502s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 17:05:07.622532175 (49h36m7.49361522s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 17:05:07.933366757 (49h36m7.804449802s) [upload=43608]: Done uploading segments... Dec 7 17:05:07.933395712 (49h36m7.804478757s) [upload=43608]: Upload finished. err: <nil> Duration: 5.804267348s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.839832722s trace: https://jaeger.dev.storj.io/trace/2dbb9d950879d488 FAIL (2025-12-07 17:17:38 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 17:04:40.750068351 (49h35m40.621151396s) [upload=43608]: Starting upload Dec 7 17:04:40.750151529 (49h35m40.621234574s) [upload=43608]: Done starting upload Dec 7 17:04:40.750252862 (49h35m40.621335927s) [upload=43608]: Uploading segments... Dec 7 17:04:40.750449186 (49h35m40.621532241s) [upload=43608]: Got next segment. Inline: false Dec 7 17:04:40.750495423 (49h35m40.621578459s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 17:04:40.750502757 (49h35m40.621585802s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 17:04:41.012862673 (49h35m40.883945718s) [seg_pos={0 0}, upload=43608]: Begin upload segment... Dec 7 17:04:41.012921534 (49h35m40.88400458s) [seg_pos={0 0}, upload=43608]: Done begin upload segment. Dec 7 17:04:41.012952173 (49h35m40.884035218s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.014639106 (49h35m40.885722151s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.015553246 (49h35m40.886636291s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016060653 (49h35m40.887143698s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016721020 (49h35m40.887804065s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.018348218 (49h35m40.889431273s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.020177682 (49h35m40.891260727s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.021303415 (49h35m40.89238646s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.023085799 (49h35m40.894168844s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.024447463 (49h35m40.895530498s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.025964030 (49h35m40.897047065s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.027289063 (49h35m40.898372108s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.028891215 (49h35m40.89997425s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.031185423 (49h35m40.902268468s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.032762747 (49h35m40.903845782s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.034246312 (49h35m40.905329347s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:42.710512985 (49h35m42.58159602s) [upload=43608]: Next returned nil segment Dec 7 17:04:42.710525298 (49h35m42.581608343s) [upload=43608]: Waiting for error group managing segments... Dec 7 17:04:43.179511201 (49h35m43.050594246s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:46.586841777 (49h35m46.457924822s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:53.671123584 (49h35m53.542206629s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:55.169666550 (49h35m55.040749595s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:57.066377404 (49h35m56.937460449s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:00.087003610 (49h35m59.958086645s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.688144215 (49h36m2.55922726s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.873597685 (49h36m2.74468073s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:03.747085322 (49h36m3.618168377s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.570204756 (49h36m4.441287801s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.766016562 (49h36m4.637099607s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:06.426653835 (49h36m6.29773689s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621680284 (49h36m7.492763339s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621710732 (49h36m7.492793777s) [seg_pos={0 0}, upload=43608]: Segment reached optimal threshold of 13 pieces. Dec 7 17:05:07.622110042 (49h36m7.493193097s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection Dec 7 17:05:07.622121895 (49h36m7.49320494s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622169114 (49h36m7.49325216s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: context canceled; context canceled Dec 7 17:05:07.622181257 (49h36m7.493264302s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622414712 (49h36m7.493497757s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: read tcp 188.166.225.170:41952->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 17:05:07.622423679 (49h36m7.493506724s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622449889 (49h36m7.493532924s) [seg_pos={0 0}, upload=43608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 17:05:07.622527457 (49h36m7.493610502s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 17:05:07.622532175 (49h36m7.49361522s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 17:05:07.933366757 (49h36m7.804449802s) [upload=43608]: Done uploading segments... Dec 7 17:05:07.933395712 (49h36m7.804478757s) [upload=43608]: Upload finished. err: <nil> Duration: 5.839858222s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.717302038s trace: https://jaeger.dev.storj.io/trace/2dbb9d950879d488 FAIL (2025-12-07 17:16:55 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 17:04:40.750068351 (49h35m40.621151396s) [upload=43608]: Starting upload Dec 7 17:04:40.750151529 (49h35m40.621234574s) [upload=43608]: Done starting upload Dec 7 17:04:40.750252862 (49h35m40.621335927s) [upload=43608]: Uploading segments... Dec 7 17:04:40.750449186 (49h35m40.621532241s) [upload=43608]: Got next segment. Inline: false Dec 7 17:04:40.750495423 (49h35m40.621578459s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 17:04:40.750502757 (49h35m40.621585802s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 17:04:41.012862673 (49h35m40.883945718s) [seg_pos={0 0}, upload=43608]: Begin upload segment... Dec 7 17:04:41.012921534 (49h35m40.88400458s) [seg_pos={0 0}, upload=43608]: Done begin upload segment. Dec 7 17:04:41.012952173 (49h35m40.884035218s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.014639106 (49h35m40.885722151s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.015553246 (49h35m40.886636291s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016060653 (49h35m40.887143698s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016721020 (49h35m40.887804065s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.018348218 (49h35m40.889431273s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.020177682 (49h35m40.891260727s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.021303415 (49h35m40.89238646s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.023085799 (49h35m40.894168844s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.024447463 (49h35m40.895530498s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.025964030 (49h35m40.897047065s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.027289063 (49h35m40.898372108s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.028891215 (49h35m40.89997425s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.031185423 (49h35m40.902268468s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.032762747 (49h35m40.903845782s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.034246312 (49h35m40.905329347s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:42.710512985 (49h35m42.58159602s) [upload=43608]: Next returned nil segment Dec 7 17:04:42.710525298 (49h35m42.581608343s) [upload=43608]: Waiting for error group managing segments... Dec 7 17:04:43.179511201 (49h35m43.050594246s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:46.586841777 (49h35m46.457924822s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:53.671123584 (49h35m53.542206629s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:55.169666550 (49h35m55.040749595s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:57.066377404 (49h35m56.937460449s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:00.087003610 (49h35m59.958086645s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.688144215 (49h36m2.55922726s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.873597685 (49h36m2.74468073s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:03.747085322 (49h36m3.618168377s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.570204756 (49h36m4.441287801s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.766016562 (49h36m4.637099607s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:06.426653835 (49h36m6.29773689s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621680284 (49h36m7.492763339s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621710732 (49h36m7.492793777s) [seg_pos={0 0}, upload=43608]: Segment reached optimal threshold of 13 pieces. Dec 7 17:05:07.622110042 (49h36m7.493193097s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection Dec 7 17:05:07.622121895 (49h36m7.49320494s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622169114 (49h36m7.49325216s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: context canceled; context canceled Dec 7 17:05:07.622181257 (49h36m7.493264302s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622414712 (49h36m7.493497757s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: read tcp 188.166.225.170:41952->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 17:05:07.622423679 (49h36m7.493506724s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622449889 (49h36m7.493532924s) [seg_pos={0 0}, upload=43608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 17:05:07.622527457 (49h36m7.493610502s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 17:05:07.622532175 (49h36m7.49361522s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 17:05:07.933366757 (49h36m7.804449802s) [upload=43608]: Done uploading segments... Dec 7 17:05:07.933395712 (49h36m7.804478757s) [upload=43608]: Upload finished. err: <nil> Duration: 5.717329621s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.627269872s trace: https://jaeger.dev.storj.io/trace/2dbb9d950879d488 FAIL (2025-12-07 17:16:21 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 17:04:40.750068351 (49h35m40.621151396s) [upload=43608]: Starting upload Dec 7 17:04:40.750151529 (49h35m40.621234574s) [upload=43608]: Done starting upload Dec 7 17:04:40.750252862 (49h35m40.621335927s) [upload=43608]: Uploading segments... Dec 7 17:04:40.750449186 (49h35m40.621532241s) [upload=43608]: Got next segment. Inline: false Dec 7 17:04:40.750495423 (49h35m40.621578459s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 17:04:40.750502757 (49h35m40.621585802s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 17:04:41.012862673 (49h35m40.883945718s) [seg_pos={0 0}, upload=43608]: Begin upload segment... Dec 7 17:04:41.012921534 (49h35m40.88400458s) [seg_pos={0 0}, upload=43608]: Done begin upload segment. Dec 7 17:04:41.012952173 (49h35m40.884035218s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.014639106 (49h35m40.885722151s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.015553246 (49h35m40.886636291s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016060653 (49h35m40.887143698s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016721020 (49h35m40.887804065s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.018348218 (49h35m40.889431273s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.020177682 (49h35m40.891260727s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.021303415 (49h35m40.89238646s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.023085799 (49h35m40.894168844s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.024447463 (49h35m40.895530498s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.025964030 (49h35m40.897047065s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.027289063 (49h35m40.898372108s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.028891215 (49h35m40.89997425s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.031185423 (49h35m40.902268468s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.032762747 (49h35m40.903845782s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.034246312 (49h35m40.905329347s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:42.710512985 (49h35m42.58159602s) [upload=43608]: Next returned nil segment Dec 7 17:04:42.710525298 (49h35m42.581608343s) [upload=43608]: Waiting for error group managing segments... Dec 7 17:04:43.179511201 (49h35m43.050594246s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:46.586841777 (49h35m46.457924822s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:53.671123584 (49h35m53.542206629s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:55.169666550 (49h35m55.040749595s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:57.066377404 (49h35m56.937460449s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:00.087003610 (49h35m59.958086645s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.688144215 (49h36m2.55922726s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.873597685 (49h36m2.74468073s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:03.747085322 (49h36m3.618168377s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.570204756 (49h36m4.441287801s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.766016562 (49h36m4.637099607s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:06.426653835 (49h36m6.29773689s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621680284 (49h36m7.492763339s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621710732 (49h36m7.492793777s) [seg_pos={0 0}, upload=43608]: Segment reached optimal threshold of 13 pieces. Dec 7 17:05:07.622110042 (49h36m7.493193097s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection Dec 7 17:05:07.622121895 (49h36m7.49320494s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622169114 (49h36m7.49325216s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: context canceled; context canceled Dec 7 17:05:07.622181257 (49h36m7.493264302s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622414712 (49h36m7.493497757s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: read tcp 188.166.225.170:41952->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 17:05:07.622423679 (49h36m7.493506724s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622449889 (49h36m7.493532924s) [seg_pos={0 0}, upload=43608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 17:05:07.622527457 (49h36m7.493610502s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 17:05:07.622532175 (49h36m7.49361522s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 17:05:07.933366757 (49h36m7.804449802s) [upload=43608]: Done uploading segments... Dec 7 17:05:07.933395712 (49h36m7.804478757s) [upload=43608]: Upload finished. err: <nil> Duration: 5.627455838s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.693768796s trace: https://jaeger.dev.storj.io/trace/2dbb9d950879d488 FAIL (2025-12-07 17:16:00 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 17:04:40.750068351 (49h35m40.621151396s) [upload=43608]: Starting upload Dec 7 17:04:40.750151529 (49h35m40.621234574s) [upload=43608]: Done starting upload Dec 7 17:04:40.750252862 (49h35m40.621335927s) [upload=43608]: Uploading segments... Dec 7 17:04:40.750449186 (49h35m40.621532241s) [upload=43608]: Got next segment. Inline: false Dec 7 17:04:40.750495423 (49h35m40.621578459s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 17:04:40.750502757 (49h35m40.621585802s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 17:04:41.012862673 (49h35m40.883945718s) [seg_pos={0 0}, upload=43608]: Begin upload segment... Dec 7 17:04:41.012921534 (49h35m40.88400458s) [seg_pos={0 0}, upload=43608]: Done begin upload segment. Dec 7 17:04:41.012952173 (49h35m40.884035218s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.014639106 (49h35m40.885722151s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.015553246 (49h35m40.886636291s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016060653 (49h35m40.887143698s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016721020 (49h35m40.887804065s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.018348218 (49h35m40.889431273s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.020177682 (49h35m40.891260727s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.021303415 (49h35m40.89238646s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.023085799 (49h35m40.894168844s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.024447463 (49h35m40.895530498s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.025964030 (49h35m40.897047065s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.027289063 (49h35m40.898372108s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.028891215 (49h35m40.89997425s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.031185423 (49h35m40.902268468s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.032762747 (49h35m40.903845782s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.034246312 (49h35m40.905329347s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:42.710512985 (49h35m42.58159602s) [upload=43608]: Next returned nil segment Dec 7 17:04:42.710525298 (49h35m42.581608343s) [upload=43608]: Waiting for error group managing segments... Dec 7 17:04:43.179511201 (49h35m43.050594246s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:46.586841777 (49h35m46.457924822s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:53.671123584 (49h35m53.542206629s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:55.169666550 (49h35m55.040749595s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:57.066377404 (49h35m56.937460449s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:00.087003610 (49h35m59.958086645s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.688144215 (49h36m2.55922726s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.873597685 (49h36m2.74468073s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:03.747085322 (49h36m3.618168377s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.570204756 (49h36m4.441287801s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.766016562 (49h36m4.637099607s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:06.426653835 (49h36m6.29773689s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621680284 (49h36m7.492763339s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621710732 (49h36m7.492793777s) [seg_pos={0 0}, upload=43608]: Segment reached optimal threshold of 13 pieces. Dec 7 17:05:07.622110042 (49h36m7.493193097s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection Dec 7 17:05:07.622121895 (49h36m7.49320494s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622169114 (49h36m7.49325216s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: context canceled; context canceled Dec 7 17:05:07.622181257 (49h36m7.493264302s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622414712 (49h36m7.493497757s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: read tcp 188.166.225.170:41952->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 17:05:07.622423679 (49h36m7.493506724s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622449889 (49h36m7.493532924s) [seg_pos={0 0}, upload=43608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 17:05:07.622527457 (49h36m7.493610502s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 17:05:07.622532175 (49h36m7.49361522s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 17:05:07.933366757 (49h36m7.804449802s) [upload=43608]: Done uploading segments... Dec 7 17:05:07.933395712 (49h36m7.804478757s) [upload=43608]: Upload finished. err: <nil> Duration: 5.693855911s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.821508078s trace: https://jaeger.dev.storj.io/trace/2dbb9d950879d488 FAIL (2025-12-07 17:15:22 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 17:04:40.750068351 (49h35m40.621151396s) [upload=43608]: Starting upload Dec 7 17:04:40.750151529 (49h35m40.621234574s) [upload=43608]: Done starting upload Dec 7 17:04:40.750252862 (49h35m40.621335927s) [upload=43608]: Uploading segments... Dec 7 17:04:40.750449186 (49h35m40.621532241s) [upload=43608]: Got next segment. Inline: false Dec 7 17:04:40.750495423 (49h35m40.621578459s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 17:04:40.750502757 (49h35m40.621585802s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 17:04:41.012862673 (49h35m40.883945718s) [seg_pos={0 0}, upload=43608]: Begin upload segment... Dec 7 17:04:41.012921534 (49h35m40.88400458s) [seg_pos={0 0}, upload=43608]: Done begin upload segment. Dec 7 17:04:41.012952173 (49h35m40.884035218s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.014639106 (49h35m40.885722151s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.015553246 (49h35m40.886636291s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016060653 (49h35m40.887143698s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016721020 (49h35m40.887804065s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.018348218 (49h35m40.889431273s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.020177682 (49h35m40.891260727s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.021303415 (49h35m40.89238646s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.023085799 (49h35m40.894168844s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.024447463 (49h35m40.895530498s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.025964030 (49h35m40.897047065s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.027289063 (49h35m40.898372108s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.028891215 (49h35m40.89997425s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.031185423 (49h35m40.902268468s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.032762747 (49h35m40.903845782s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.034246312 (49h35m40.905329347s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:42.710512985 (49h35m42.58159602s) [upload=43608]: Next returned nil segment Dec 7 17:04:42.710525298 (49h35m42.581608343s) [upload=43608]: Waiting for error group managing segments... Dec 7 17:04:43.179511201 (49h35m43.050594246s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:46.586841777 (49h35m46.457924822s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:53.671123584 (49h35m53.542206629s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:55.169666550 (49h35m55.040749595s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:57.066377404 (49h35m56.937460449s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:00.087003610 (49h35m59.958086645s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.688144215 (49h36m2.55922726s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.873597685 (49h36m2.74468073s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:03.747085322 (49h36m3.618168377s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.570204756 (49h36m4.441287801s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.766016562 (49h36m4.637099607s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:06.426653835 (49h36m6.29773689s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621680284 (49h36m7.492763339s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621710732 (49h36m7.492793777s) [seg_pos={0 0}, upload=43608]: Segment reached optimal threshold of 13 pieces. Dec 7 17:05:07.622110042 (49h36m7.493193097s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection Dec 7 17:05:07.622121895 (49h36m7.49320494s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622169114 (49h36m7.49325216s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: context canceled; context canceled Dec 7 17:05:07.622181257 (49h36m7.493264302s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622414712 (49h36m7.493497757s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: read tcp 188.166.225.170:41952->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 17:05:07.622423679 (49h36m7.493506724s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622449889 (49h36m7.493532924s) [seg_pos={0 0}, upload=43608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 17:05:07.622527457 (49h36m7.493610502s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 17:05:07.622532175 (49h36m7.49361522s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 17:05:07.933366757 (49h36m7.804449802s) [upload=43608]: Done uploading segments... Dec 7 17:05:07.933395712 (49h36m7.804478757s) [upload=43608]: Upload finished. err: <nil> Duration: 5.82154073s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.886752405s trace: https://jaeger.dev.storj.io/trace/2dbb9d950879d488 FAIL (2025-12-07 17:14:52 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 17:04:40.750068351 (49h35m40.621151396s) [upload=43608]: Starting upload Dec 7 17:04:40.750151529 (49h35m40.621234574s) [upload=43608]: Done starting upload Dec 7 17:04:40.750252862 (49h35m40.621335927s) [upload=43608]: Uploading segments... Dec 7 17:04:40.750449186 (49h35m40.621532241s) [upload=43608]: Got next segment. Inline: false Dec 7 17:04:40.750495423 (49h35m40.621578459s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 17:04:40.750502757 (49h35m40.621585802s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 17:04:41.012862673 (49h35m40.883945718s) [seg_pos={0 0}, upload=43608]: Begin upload segment... Dec 7 17:04:41.012921534 (49h35m40.88400458s) [seg_pos={0 0}, upload=43608]: Done begin upload segment. Dec 7 17:04:41.012952173 (49h35m40.884035218s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.014639106 (49h35m40.885722151s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.015553246 (49h35m40.886636291s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016060653 (49h35m40.887143698s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016721020 (49h35m40.887804065s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.018348218 (49h35m40.889431273s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.020177682 (49h35m40.891260727s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.021303415 (49h35m40.89238646s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.023085799 (49h35m40.894168844s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.024447463 (49h35m40.895530498s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.025964030 (49h35m40.897047065s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.027289063 (49h35m40.898372108s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.028891215 (49h35m40.89997425s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.031185423 (49h35m40.902268468s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.032762747 (49h35m40.903845782s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.034246312 (49h35m40.905329347s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:42.710512985 (49h35m42.58159602s) [upload=43608]: Next returned nil segment Dec 7 17:04:42.710525298 (49h35m42.581608343s) [upload=43608]: Waiting for error group managing segments... Dec 7 17:04:43.179511201 (49h35m43.050594246s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:46.586841777 (49h35m46.457924822s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:53.671123584 (49h35m53.542206629s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:55.169666550 (49h35m55.040749595s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:57.066377404 (49h35m56.937460449s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:00.087003610 (49h35m59.958086645s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.688144215 (49h36m2.55922726s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.873597685 (49h36m2.74468073s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:03.747085322 (49h36m3.618168377s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.570204756 (49h36m4.441287801s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.766016562 (49h36m4.637099607s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:06.426653835 (49h36m6.29773689s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621680284 (49h36m7.492763339s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621710732 (49h36m7.492793777s) [seg_pos={0 0}, upload=43608]: Segment reached optimal threshold of 13 pieces. Dec 7 17:05:07.622110042 (49h36m7.493193097s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection Dec 7 17:05:07.622121895 (49h36m7.49320494s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622169114 (49h36m7.49325216s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: context canceled; context canceled Dec 7 17:05:07.622181257 (49h36m7.493264302s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622414712 (49h36m7.493497757s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: read tcp 188.166.225.170:41952->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 17:05:07.622423679 (49h36m7.493506724s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622449889 (49h36m7.493532924s) [seg_pos={0 0}, upload=43608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 17:05:07.622527457 (49h36m7.493610502s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 17:05:07.622532175 (49h36m7.49361522s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 17:05:07.933366757 (49h36m7.804449802s) [upload=43608]: Done uploading segments... Dec 7 17:05:07.933395712 (49h36m7.804478757s) [upload=43608]: Upload finished. err: <nil> Duration: 5.886782153s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 7.08698726s trace: https://jaeger.dev.storj.io/trace/2dbb9d950879d488 FAIL (2025-12-07 17:14:22 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 17:04:40.750068351 (49h35m40.621151396s) [upload=43608]: Starting upload Dec 7 17:04:40.750151529 (49h35m40.621234574s) [upload=43608]: Done starting upload Dec 7 17:04:40.750252862 (49h35m40.621335927s) [upload=43608]: Uploading segments... Dec 7 17:04:40.750449186 (49h35m40.621532241s) [upload=43608]: Got next segment. Inline: false Dec 7 17:04:40.750495423 (49h35m40.621578459s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 17:04:40.750502757 (49h35m40.621585802s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 17:04:41.012862673 (49h35m40.883945718s) [seg_pos={0 0}, upload=43608]: Begin upload segment... Dec 7 17:04:41.012921534 (49h35m40.88400458s) [seg_pos={0 0}, upload=43608]: Done begin upload segment. Dec 7 17:04:41.012952173 (49h35m40.884035218s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.014639106 (49h35m40.885722151s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.015553246 (49h35m40.886636291s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016060653 (49h35m40.887143698s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016721020 (49h35m40.887804065s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.018348218 (49h35m40.889431273s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.020177682 (49h35m40.891260727s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.021303415 (49h35m40.89238646s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.023085799 (49h35m40.894168844s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.024447463 (49h35m40.895530498s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.025964030 (49h35m40.897047065s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.027289063 (49h35m40.898372108s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.028891215 (49h35m40.89997425s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.031185423 (49h35m40.902268468s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.032762747 (49h35m40.903845782s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.034246312 (49h35m40.905329347s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:42.710512985 (49h35m42.58159602s) [upload=43608]: Next returned nil segment Dec 7 17:04:42.710525298 (49h35m42.581608343s) [upload=43608]: Waiting for error group managing segments... Dec 7 17:04:43.179511201 (49h35m43.050594246s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:46.586841777 (49h35m46.457924822s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:53.671123584 (49h35m53.542206629s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:55.169666550 (49h35m55.040749595s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:57.066377404 (49h35m56.937460449s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:00.087003610 (49h35m59.958086645s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.688144215 (49h36m2.55922726s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.873597685 (49h36m2.74468073s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:03.747085322 (49h36m3.618168377s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.570204756 (49h36m4.441287801s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.766016562 (49h36m4.637099607s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:06.426653835 (49h36m6.29773689s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621680284 (49h36m7.492763339s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621710732 (49h36m7.492793777s) [seg_pos={0 0}, upload=43608]: Segment reached optimal threshold of 13 pieces. Dec 7 17:05:07.622110042 (49h36m7.493193097s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection Dec 7 17:05:07.622121895 (49h36m7.49320494s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622169114 (49h36m7.49325216s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: context canceled; context canceled Dec 7 17:05:07.622181257 (49h36m7.493264302s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622414712 (49h36m7.493497757s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: read tcp 188.166.225.170:41952->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 17:05:07.622423679 (49h36m7.493506724s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622449889 (49h36m7.493532924s) [seg_pos={0 0}, upload=43608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 17:05:07.622527457 (49h36m7.493610502s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 17:05:07.622532175 (49h36m7.49361522s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 17:05:07.933366757 (49h36m7.804449802s) [upload=43608]: Done uploading segments... Dec 7 17:05:07.933395712 (49h36m7.804478757s) [upload=43608]: Upload finished. err: <nil> Duration: 7.087022087s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.12312077s trace: https://jaeger.dev.storj.io/trace/2dbb9d950879d488 FAIL (2025-12-07 17:13:45 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 17:04:40.750068351 (49h35m40.621151396s) [upload=43608]: Starting upload Dec 7 17:04:40.750151529 (49h35m40.621234574s) [upload=43608]: Done starting upload Dec 7 17:04:40.750252862 (49h35m40.621335927s) [upload=43608]: Uploading segments... Dec 7 17:04:40.750449186 (49h35m40.621532241s) [upload=43608]: Got next segment. Inline: false Dec 7 17:04:40.750495423 (49h35m40.621578459s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 17:04:40.750502757 (49h35m40.621585802s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 17:04:41.012862673 (49h35m40.883945718s) [seg_pos={0 0}, upload=43608]: Begin upload segment... Dec 7 17:04:41.012921534 (49h35m40.88400458s) [seg_pos={0 0}, upload=43608]: Done begin upload segment. Dec 7 17:04:41.012952173 (49h35m40.884035218s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.014639106 (49h35m40.885722151s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.015553246 (49h35m40.886636291s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016060653 (49h35m40.887143698s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016721020 (49h35m40.887804065s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.018348218 (49h35m40.889431273s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.020177682 (49h35m40.891260727s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.021303415 (49h35m40.89238646s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.023085799 (49h35m40.894168844s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.024447463 (49h35m40.895530498s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.025964030 (49h35m40.897047065s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.027289063 (49h35m40.898372108s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.028891215 (49h35m40.89997425s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.031185423 (49h35m40.902268468s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.032762747 (49h35m40.903845782s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.034246312 (49h35m40.905329347s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:42.710512985 (49h35m42.58159602s) [upload=43608]: Next returned nil segment Dec 7 17:04:42.710525298 (49h35m42.581608343s) [upload=43608]: Waiting for error group managing segments... Dec 7 17:04:43.179511201 (49h35m43.050594246s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:46.586841777 (49h35m46.457924822s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:53.671123584 (49h35m53.542206629s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:55.169666550 (49h35m55.040749595s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:57.066377404 (49h35m56.937460449s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:00.087003610 (49h35m59.958086645s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.688144215 (49h36m2.55922726s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.873597685 (49h36m2.74468073s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:03.747085322 (49h36m3.618168377s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.570204756 (49h36m4.441287801s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.766016562 (49h36m4.637099607s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:06.426653835 (49h36m6.29773689s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621680284 (49h36m7.492763339s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621710732 (49h36m7.492793777s) [seg_pos={0 0}, upload=43608]: Segment reached optimal threshold of 13 pieces. Dec 7 17:05:07.622110042 (49h36m7.493193097s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection Dec 7 17:05:07.622121895 (49h36m7.49320494s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622169114 (49h36m7.49325216s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: context canceled; context canceled Dec 7 17:05:07.622181257 (49h36m7.493264302s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622414712 (49h36m7.493497757s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: read tcp 188.166.225.170:41952->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 17:05:07.622423679 (49h36m7.493506724s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622449889 (49h36m7.493532924s) [seg_pos={0 0}, upload=43608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 17:05:07.622527457 (49h36m7.493610502s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 17:05:07.622532175 (49h36m7.49361522s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 17:05:07.933366757 (49h36m7.804449802s) [upload=43608]: Done uploading segments... Dec 7 17:05:07.933395712 (49h36m7.804478757s) [upload=43608]: Upload finished. err: <nil> Duration: 6.12315717s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.885147471s trace: https://jaeger.dev.storj.io/trace/2dbb9d950879d488 FAIL (2025-12-07 17:13:07 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 17:04:40.750068351 (49h35m40.621151396s) [upload=43608]: Starting upload Dec 7 17:04:40.750151529 (49h35m40.621234574s) [upload=43608]: Done starting upload Dec 7 17:04:40.750252862 (49h35m40.621335927s) [upload=43608]: Uploading segments... Dec 7 17:04:40.750449186 (49h35m40.621532241s) [upload=43608]: Got next segment. Inline: false Dec 7 17:04:40.750495423 (49h35m40.621578459s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 17:04:40.750502757 (49h35m40.621585802s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 17:04:41.012862673 (49h35m40.883945718s) [seg_pos={0 0}, upload=43608]: Begin upload segment... Dec 7 17:04:41.012921534 (49h35m40.88400458s) [seg_pos={0 0}, upload=43608]: Done begin upload segment. Dec 7 17:04:41.012952173 (49h35m40.884035218s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.014639106 (49h35m40.885722151s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.015553246 (49h35m40.886636291s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016060653 (49h35m40.887143698s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016721020 (49h35m40.887804065s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.018348218 (49h35m40.889431273s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.020177682 (49h35m40.891260727s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.021303415 (49h35m40.89238646s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.023085799 (49h35m40.894168844s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.024447463 (49h35m40.895530498s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.025964030 (49h35m40.897047065s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.027289063 (49h35m40.898372108s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.028891215 (49h35m40.89997425s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.031185423 (49h35m40.902268468s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.032762747 (49h35m40.903845782s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.034246312 (49h35m40.905329347s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:42.710512985 (49h35m42.58159602s) [upload=43608]: Next returned nil segment Dec 7 17:04:42.710525298 (49h35m42.581608343s) [upload=43608]: Waiting for error group managing segments... Dec 7 17:04:43.179511201 (49h35m43.050594246s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:46.586841777 (49h35m46.457924822s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:53.671123584 (49h35m53.542206629s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:55.169666550 (49h35m55.040749595s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:57.066377404 (49h35m56.937460449s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:00.087003610 (49h35m59.958086645s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.688144215 (49h36m2.55922726s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.873597685 (49h36m2.74468073s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:03.747085322 (49h36m3.618168377s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.570204756 (49h36m4.441287801s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.766016562 (49h36m4.637099607s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:06.426653835 (49h36m6.29773689s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621680284 (49h36m7.492763339s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621710732 (49h36m7.492793777s) [seg_pos={0 0}, upload=43608]: Segment reached optimal threshold of 13 pieces. Dec 7 17:05:07.622110042 (49h36m7.493193097s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection Dec 7 17:05:07.622121895 (49h36m7.49320494s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622169114 (49h36m7.49325216s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: context canceled; context canceled Dec 7 17:05:07.622181257 (49h36m7.493264302s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622414712 (49h36m7.493497757s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: read tcp 188.166.225.170:41952->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 17:05:07.622423679 (49h36m7.493506724s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622449889 (49h36m7.493532924s) [seg_pos={0 0}, upload=43608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 17:05:07.622527457 (49h36m7.493610502s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 17:05:07.622532175 (49h36m7.49361522s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 17:05:07.933366757 (49h36m7.804449802s) [upload=43608]: Done uploading segments... Dec 7 17:05:07.933395712 (49h36m7.804478757s) [upload=43608]: Upload finished. err: <nil> Duration: 5.885176365s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.980496665s trace: https://jaeger.dev.storj.io/trace/2dbb9d950879d488 FAIL (2025-12-07 17:12:46 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 17:04:40.750068351 (49h35m40.621151396s) [upload=43608]: Starting upload Dec 7 17:04:40.750151529 (49h35m40.621234574s) [upload=43608]: Done starting upload Dec 7 17:04:40.750252862 (49h35m40.621335927s) [upload=43608]: Uploading segments... Dec 7 17:04:40.750449186 (49h35m40.621532241s) [upload=43608]: Got next segment. Inline: false Dec 7 17:04:40.750495423 (49h35m40.621578459s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 17:04:40.750502757 (49h35m40.621585802s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 17:04:41.012862673 (49h35m40.883945718s) [seg_pos={0 0}, upload=43608]: Begin upload segment... Dec 7 17:04:41.012921534 (49h35m40.88400458s) [seg_pos={0 0}, upload=43608]: Done begin upload segment. Dec 7 17:04:41.012952173 (49h35m40.884035218s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.014639106 (49h35m40.885722151s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.015553246 (49h35m40.886636291s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016060653 (49h35m40.887143698s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016721020 (49h35m40.887804065s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.018348218 (49h35m40.889431273s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.020177682 (49h35m40.891260727s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.021303415 (49h35m40.89238646s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.023085799 (49h35m40.894168844s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.024447463 (49h35m40.895530498s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.025964030 (49h35m40.897047065s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.027289063 (49h35m40.898372108s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.028891215 (49h35m40.89997425s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.031185423 (49h35m40.902268468s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.032762747 (49h35m40.903845782s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.034246312 (49h35m40.905329347s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:42.710512985 (49h35m42.58159602s) [upload=43608]: Next returned nil segment Dec 7 17:04:42.710525298 (49h35m42.581608343s) [upload=43608]: Waiting for error group managing segments... Dec 7 17:04:43.179511201 (49h35m43.050594246s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:46.586841777 (49h35m46.457924822s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:53.671123584 (49h35m53.542206629s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:55.169666550 (49h35m55.040749595s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:57.066377404 (49h35m56.937460449s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:00.087003610 (49h35m59.958086645s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.688144215 (49h36m2.55922726s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.873597685 (49h36m2.74468073s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:03.747085322 (49h36m3.618168377s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.570204756 (49h36m4.441287801s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.766016562 (49h36m4.637099607s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:06.426653835 (49h36m6.29773689s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621680284 (49h36m7.492763339s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621710732 (49h36m7.492793777s) [seg_pos={0 0}, upload=43608]: Segment reached optimal threshold of 13 pieces. Dec 7 17:05:07.622110042 (49h36m7.493193097s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection Dec 7 17:05:07.622121895 (49h36m7.49320494s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622169114 (49h36m7.49325216s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: context canceled; context canceled Dec 7 17:05:07.622181257 (49h36m7.493264302s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622414712 (49h36m7.493497757s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: read tcp 188.166.225.170:41952->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 17:05:07.622423679 (49h36m7.493506724s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622449889 (49h36m7.493532924s) [seg_pos={0 0}, upload=43608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 17:05:07.622527457 (49h36m7.493610502s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 17:05:07.622532175 (49h36m7.49361522s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 17:05:07.933366757 (49h36m7.804449802s) [upload=43608]: Done uploading segments... Dec 7 17:05:07.933395712 (49h36m7.804478757s) [upload=43608]: Upload finished. err: <nil> Duration: 5.980524097s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.749849915s trace: https://jaeger.dev.storj.io/trace/2dbb9d950879d488 FAIL (2025-12-07 17:11:50 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 17:04:40.750068351 (49h35m40.621151396s) [upload=43608]: Starting upload Dec 7 17:04:40.750151529 (49h35m40.621234574s) [upload=43608]: Done starting upload Dec 7 17:04:40.750252862 (49h35m40.621335927s) [upload=43608]: Uploading segments... Dec 7 17:04:40.750449186 (49h35m40.621532241s) [upload=43608]: Got next segment. Inline: false Dec 7 17:04:40.750495423 (49h35m40.621578459s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 17:04:40.750502757 (49h35m40.621585802s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 17:04:41.012862673 (49h35m40.883945718s) [seg_pos={0 0}, upload=43608]: Begin upload segment... Dec 7 17:04:41.012921534 (49h35m40.88400458s) [seg_pos={0 0}, upload=43608]: Done begin upload segment. Dec 7 17:04:41.012952173 (49h35m40.884035218s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.014639106 (49h35m40.885722151s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.015553246 (49h35m40.886636291s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016060653 (49h35m40.887143698s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016721020 (49h35m40.887804065s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.018348218 (49h35m40.889431273s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.020177682 (49h35m40.891260727s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.021303415 (49h35m40.89238646s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.023085799 (49h35m40.894168844s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.024447463 (49h35m40.895530498s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.025964030 (49h35m40.897047065s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.027289063 (49h35m40.898372108s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.028891215 (49h35m40.89997425s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.031185423 (49h35m40.902268468s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.032762747 (49h35m40.903845782s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.034246312 (49h35m40.905329347s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:42.710512985 (49h35m42.58159602s) [upload=43608]: Next returned nil segment Dec 7 17:04:42.710525298 (49h35m42.581608343s) [upload=43608]: Waiting for error group managing segments... Dec 7 17:04:43.179511201 (49h35m43.050594246s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:46.586841777 (49h35m46.457924822s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:53.671123584 (49h35m53.542206629s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:55.169666550 (49h35m55.040749595s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:57.066377404 (49h35m56.937460449s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:00.087003610 (49h35m59.958086645s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.688144215 (49h36m2.55922726s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.873597685 (49h36m2.74468073s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:03.747085322 (49h36m3.618168377s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.570204756 (49h36m4.441287801s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.766016562 (49h36m4.637099607s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:06.426653835 (49h36m6.29773689s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621680284 (49h36m7.492763339s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621710732 (49h36m7.492793777s) [seg_pos={0 0}, upload=43608]: Segment reached optimal threshold of 13 pieces. Dec 7 17:05:07.622110042 (49h36m7.493193097s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection Dec 7 17:05:07.622121895 (49h36m7.49320494s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622169114 (49h36m7.49325216s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: context canceled; context canceled Dec 7 17:05:07.622181257 (49h36m7.493264302s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622414712 (49h36m7.493497757s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: read tcp 188.166.225.170:41952->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 17:05:07.622423679 (49h36m7.493506724s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622449889 (49h36m7.493532924s) [seg_pos={0 0}, upload=43608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 17:05:07.622527457 (49h36m7.493610502s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 17:05:07.622532175 (49h36m7.49361522s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 17:05:07.933366757 (49h36m7.804449802s) [upload=43608]: Done uploading segments... Dec 7 17:05:07.933395712 (49h36m7.804478757s) [upload=43608]: Upload finished. err: <nil> Duration: 5.749879192s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.716667743s trace: https://jaeger.dev.storj.io/trace/2dbb9d950879d488 FAIL (2025-12-07 17:10:50 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 17:04:40.750068351 (49h35m40.621151396s) [upload=43608]: Starting upload Dec 7 17:04:40.750151529 (49h35m40.621234574s) [upload=43608]: Done starting upload Dec 7 17:04:40.750252862 (49h35m40.621335927s) [upload=43608]: Uploading segments... Dec 7 17:04:40.750449186 (49h35m40.621532241s) [upload=43608]: Got next segment. Inline: false Dec 7 17:04:40.750495423 (49h35m40.621578459s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 17:04:40.750502757 (49h35m40.621585802s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 17:04:41.012862673 (49h35m40.883945718s) [seg_pos={0 0}, upload=43608]: Begin upload segment... Dec 7 17:04:41.012921534 (49h35m40.88400458s) [seg_pos={0 0}, upload=43608]: Done begin upload segment. Dec 7 17:04:41.012952173 (49h35m40.884035218s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.014639106 (49h35m40.885722151s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.015553246 (49h35m40.886636291s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016060653 (49h35m40.887143698s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016721020 (49h35m40.887804065s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.018348218 (49h35m40.889431273s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.020177682 (49h35m40.891260727s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.021303415 (49h35m40.89238646s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.023085799 (49h35m40.894168844s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.024447463 (49h35m40.895530498s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.025964030 (49h35m40.897047065s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.027289063 (49h35m40.898372108s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.028891215 (49h35m40.89997425s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.031185423 (49h35m40.902268468s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.032762747 (49h35m40.903845782s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.034246312 (49h35m40.905329347s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:42.710512985 (49h35m42.58159602s) [upload=43608]: Next returned nil segment Dec 7 17:04:42.710525298 (49h35m42.581608343s) [upload=43608]: Waiting for error group managing segments... Dec 7 17:04:43.179511201 (49h35m43.050594246s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:46.586841777 (49h35m46.457924822s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:53.671123584 (49h35m53.542206629s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:55.169666550 (49h35m55.040749595s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:57.066377404 (49h35m56.937460449s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:00.087003610 (49h35m59.958086645s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.688144215 (49h36m2.55922726s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.873597685 (49h36m2.74468073s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:03.747085322 (49h36m3.618168377s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.570204756 (49h36m4.441287801s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.766016562 (49h36m4.637099607s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:06.426653835 (49h36m6.29773689s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621680284 (49h36m7.492763339s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621710732 (49h36m7.492793777s) [seg_pos={0 0}, upload=43608]: Segment reached optimal threshold of 13 pieces. Dec 7 17:05:07.622110042 (49h36m7.493193097s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection Dec 7 17:05:07.622121895 (49h36m7.49320494s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622169114 (49h36m7.49325216s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: context canceled; context canceled Dec 7 17:05:07.622181257 (49h36m7.493264302s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622414712 (49h36m7.493497757s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: read tcp 188.166.225.170:41952->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 17:05:07.622423679 (49h36m7.493506724s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622449889 (49h36m7.493532924s) [seg_pos={0 0}, upload=43608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 17:05:07.622527457 (49h36m7.493610502s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 17:05:07.622532175 (49h36m7.49361522s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 17:05:07.933366757 (49h36m7.804449802s) [upload=43608]: Done uploading segments... Dec 7 17:05:07.933395712 (49h36m7.804478757s) [upload=43608]: Upload finished. err: <nil> Duration: 5.716695656s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.676069275s trace: https://jaeger.dev.storj.io/trace/2dbb9d950879d488 FAIL (2025-12-07 17:10:23 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 17:04:40.750068351 (49h35m40.621151396s) [upload=43608]: Starting upload Dec 7 17:04:40.750151529 (49h35m40.621234574s) [upload=43608]: Done starting upload Dec 7 17:04:40.750252862 (49h35m40.621335927s) [upload=43608]: Uploading segments... Dec 7 17:04:40.750449186 (49h35m40.621532241s) [upload=43608]: Got next segment. Inline: false Dec 7 17:04:40.750495423 (49h35m40.621578459s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 17:04:40.750502757 (49h35m40.621585802s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 17:04:41.012862673 (49h35m40.883945718s) [seg_pos={0 0}, upload=43608]: Begin upload segment... Dec 7 17:04:41.012921534 (49h35m40.88400458s) [seg_pos={0 0}, upload=43608]: Done begin upload segment. Dec 7 17:04:41.012952173 (49h35m40.884035218s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.014639106 (49h35m40.885722151s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.015553246 (49h35m40.886636291s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016060653 (49h35m40.887143698s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016721020 (49h35m40.887804065s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.018348218 (49h35m40.889431273s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.020177682 (49h35m40.891260727s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.021303415 (49h35m40.89238646s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.023085799 (49h35m40.894168844s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.024447463 (49h35m40.895530498s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.025964030 (49h35m40.897047065s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.027289063 (49h35m40.898372108s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.028891215 (49h35m40.89997425s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.031185423 (49h35m40.902268468s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.032762747 (49h35m40.903845782s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.034246312 (49h35m40.905329347s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:42.710512985 (49h35m42.58159602s) [upload=43608]: Next returned nil segment Dec 7 17:04:42.710525298 (49h35m42.581608343s) [upload=43608]: Waiting for error group managing segments... Dec 7 17:04:43.179511201 (49h35m43.050594246s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:46.586841777 (49h35m46.457924822s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:53.671123584 (49h35m53.542206629s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:55.169666550 (49h35m55.040749595s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:57.066377404 (49h35m56.937460449s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:00.087003610 (49h35m59.958086645s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.688144215 (49h36m2.55922726s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.873597685 (49h36m2.74468073s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:03.747085322 (49h36m3.618168377s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.570204756 (49h36m4.441287801s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.766016562 (49h36m4.637099607s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:06.426653835 (49h36m6.29773689s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621680284 (49h36m7.492763339s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621710732 (49h36m7.492793777s) [seg_pos={0 0}, upload=43608]: Segment reached optimal threshold of 13 pieces. Dec 7 17:05:07.622110042 (49h36m7.493193097s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection Dec 7 17:05:07.622121895 (49h36m7.49320494s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622169114 (49h36m7.49325216s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: context canceled; context canceled Dec 7 17:05:07.622181257 (49h36m7.493264302s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622414712 (49h36m7.493497757s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: read tcp 188.166.225.170:41952->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 17:05:07.622423679 (49h36m7.493506724s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622449889 (49h36m7.493532924s) [seg_pos={0 0}, upload=43608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 17:05:07.622527457 (49h36m7.493610502s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 17:05:07.622532175 (49h36m7.49361522s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 17:05:07.933366757 (49h36m7.804449802s) [upload=43608]: Done uploading segments... Dec 7 17:05:07.933395712 (49h36m7.804478757s) [upload=43608]: Upload finished. err: <nil> Duration: 5.67615525s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.786716719s trace: https://jaeger.dev.storj.io/trace/2dbb9d950879d488 FAIL (2025-12-07 17:09:10 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 17:04:40.750068351 (49h35m40.621151396s) [upload=43608]: Starting upload Dec 7 17:04:40.750151529 (49h35m40.621234574s) [upload=43608]: Done starting upload Dec 7 17:04:40.750252862 (49h35m40.621335927s) [upload=43608]: Uploading segments... Dec 7 17:04:40.750449186 (49h35m40.621532241s) [upload=43608]: Got next segment. Inline: false Dec 7 17:04:40.750495423 (49h35m40.621578459s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 17:04:40.750502757 (49h35m40.621585802s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 17:04:41.012862673 (49h35m40.883945718s) [seg_pos={0 0}, upload=43608]: Begin upload segment... Dec 7 17:04:41.012921534 (49h35m40.88400458s) [seg_pos={0 0}, upload=43608]: Done begin upload segment. Dec 7 17:04:41.012952173 (49h35m40.884035218s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.014639106 (49h35m40.885722151s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.015553246 (49h35m40.886636291s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016060653 (49h35m40.887143698s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016721020 (49h35m40.887804065s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.018348218 (49h35m40.889431273s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.020177682 (49h35m40.891260727s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.021303415 (49h35m40.89238646s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.023085799 (49h35m40.894168844s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.024447463 (49h35m40.895530498s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.025964030 (49h35m40.897047065s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.027289063 (49h35m40.898372108s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.028891215 (49h35m40.89997425s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.031185423 (49h35m40.902268468s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.032762747 (49h35m40.903845782s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.034246312 (49h35m40.905329347s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:42.710512985 (49h35m42.58159602s) [upload=43608]: Next returned nil segment Dec 7 17:04:42.710525298 (49h35m42.581608343s) [upload=43608]: Waiting for error group managing segments... Dec 7 17:04:43.179511201 (49h35m43.050594246s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:46.586841777 (49h35m46.457924822s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:53.671123584 (49h35m53.542206629s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:55.169666550 (49h35m55.040749595s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:57.066377404 (49h35m56.937460449s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:00.087003610 (49h35m59.958086645s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.688144215 (49h36m2.55922726s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.873597685 (49h36m2.74468073s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:03.747085322 (49h36m3.618168377s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.570204756 (49h36m4.441287801s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.766016562 (49h36m4.637099607s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:06.426653835 (49h36m6.29773689s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621680284 (49h36m7.492763339s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621710732 (49h36m7.492793777s) [seg_pos={0 0}, upload=43608]: Segment reached optimal threshold of 13 pieces. Dec 7 17:05:07.622110042 (49h36m7.493193097s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection Dec 7 17:05:07.622121895 (49h36m7.49320494s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622169114 (49h36m7.49325216s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: context canceled; context canceled Dec 7 17:05:07.622181257 (49h36m7.493264302s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622414712 (49h36m7.493497757s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: read tcp 188.166.225.170:41952->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 17:05:07.622423679 (49h36m7.493506724s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622449889 (49h36m7.493532924s) [seg_pos={0 0}, upload=43608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 17:05:07.622527457 (49h36m7.493610502s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 17:05:07.622532175 (49h36m7.49361522s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 17:05:07.933366757 (49h36m7.804449802s) [upload=43608]: Done uploading segments... Dec 7 17:05:07.933395712 (49h36m7.804478757s) [upload=43608]: Upload finished. err: <nil> Duration: 5.786750815s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.774540886s trace: https://jaeger.dev.storj.io/trace/2dbb9d950879d488 FAIL (2025-12-07 17:08:24 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 17:04:40.750068351 (49h35m40.621151396s) [upload=43608]: Starting upload Dec 7 17:04:40.750151529 (49h35m40.621234574s) [upload=43608]: Done starting upload Dec 7 17:04:40.750252862 (49h35m40.621335927s) [upload=43608]: Uploading segments... Dec 7 17:04:40.750449186 (49h35m40.621532241s) [upload=43608]: Got next segment. Inline: false Dec 7 17:04:40.750495423 (49h35m40.621578459s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 17:04:40.750502757 (49h35m40.621585802s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 17:04:41.012862673 (49h35m40.883945718s) [seg_pos={0 0}, upload=43608]: Begin upload segment... Dec 7 17:04:41.012921534 (49h35m40.88400458s) [seg_pos={0 0}, upload=43608]: Done begin upload segment. Dec 7 17:04:41.012952173 (49h35m40.884035218s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.014639106 (49h35m40.885722151s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.015553246 (49h35m40.886636291s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016060653 (49h35m40.887143698s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016721020 (49h35m40.887804065s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.018348218 (49h35m40.889431273s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.020177682 (49h35m40.891260727s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.021303415 (49h35m40.89238646s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.023085799 (49h35m40.894168844s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.024447463 (49h35m40.895530498s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.025964030 (49h35m40.897047065s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.027289063 (49h35m40.898372108s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.028891215 (49h35m40.89997425s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.031185423 (49h35m40.902268468s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.032762747 (49h35m40.903845782s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.034246312 (49h35m40.905329347s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:42.710512985 (49h35m42.58159602s) [upload=43608]: Next returned nil segment Dec 7 17:04:42.710525298 (49h35m42.581608343s) [upload=43608]: Waiting for error group managing segments... Dec 7 17:04:43.179511201 (49h35m43.050594246s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:46.586841777 (49h35m46.457924822s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:53.671123584 (49h35m53.542206629s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:55.169666550 (49h35m55.040749595s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:57.066377404 (49h35m56.937460449s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:00.087003610 (49h35m59.958086645s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.688144215 (49h36m2.55922726s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.873597685 (49h36m2.74468073s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:03.747085322 (49h36m3.618168377s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.570204756 (49h36m4.441287801s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.766016562 (49h36m4.637099607s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:06.426653835 (49h36m6.29773689s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621680284 (49h36m7.492763339s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621710732 (49h36m7.492793777s) [seg_pos={0 0}, upload=43608]: Segment reached optimal threshold of 13 pieces. Dec 7 17:05:07.622110042 (49h36m7.493193097s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection Dec 7 17:05:07.622121895 (49h36m7.49320494s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622169114 (49h36m7.49325216s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: context canceled; context canceled Dec 7 17:05:07.622181257 (49h36m7.493264302s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622414712 (49h36m7.493497757s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: read tcp 188.166.225.170:41952->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 17:05:07.622423679 (49h36m7.493506724s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622449889 (49h36m7.493532924s) [seg_pos={0 0}, upload=43608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 17:05:07.622527457 (49h36m7.493610502s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 17:05:07.622532175 (49h36m7.49361522s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 17:05:07.933366757 (49h36m7.804449802s) [upload=43608]: Done uploading segments... Dec 7 17:05:07.933395712 (49h36m7.804478757s) [upload=43608]: Upload finished. err: <nil> Duration: 5.774577577s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.672390079s trace: https://jaeger.dev.storj.io/trace/2dbb9d950879d488 FAIL (2025-12-07 17:08:01 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 17:04:40.750068351 (49h35m40.621151396s) [upload=43608]: Starting upload Dec 7 17:04:40.750151529 (49h35m40.621234574s) [upload=43608]: Done starting upload Dec 7 17:04:40.750252862 (49h35m40.621335927s) [upload=43608]: Uploading segments... Dec 7 17:04:40.750449186 (49h35m40.621532241s) [upload=43608]: Got next segment. Inline: false Dec 7 17:04:40.750495423 (49h35m40.621578459s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 17:04:40.750502757 (49h35m40.621585802s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 17:04:41.012862673 (49h35m40.883945718s) [seg_pos={0 0}, upload=43608]: Begin upload segment... Dec 7 17:04:41.012921534 (49h35m40.88400458s) [seg_pos={0 0}, upload=43608]: Done begin upload segment. Dec 7 17:04:41.012952173 (49h35m40.884035218s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.014639106 (49h35m40.885722151s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.015553246 (49h35m40.886636291s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016060653 (49h35m40.887143698s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016721020 (49h35m40.887804065s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.018348218 (49h35m40.889431273s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.020177682 (49h35m40.891260727s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.021303415 (49h35m40.89238646s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.023085799 (49h35m40.894168844s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.024447463 (49h35m40.895530498s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.025964030 (49h35m40.897047065s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.027289063 (49h35m40.898372108s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.028891215 (49h35m40.89997425s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.031185423 (49h35m40.902268468s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.032762747 (49h35m40.903845782s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.034246312 (49h35m40.905329347s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:42.710512985 (49h35m42.58159602s) [upload=43608]: Next returned nil segment Dec 7 17:04:42.710525298 (49h35m42.581608343s) [upload=43608]: Waiting for error group managing segments... Dec 7 17:04:43.179511201 (49h35m43.050594246s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:46.586841777 (49h35m46.457924822s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:53.671123584 (49h35m53.542206629s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:55.169666550 (49h35m55.040749595s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:57.066377404 (49h35m56.937460449s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:00.087003610 (49h35m59.958086645s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.688144215 (49h36m2.55922726s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.873597685 (49h36m2.74468073s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:03.747085322 (49h36m3.618168377s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.570204756 (49h36m4.441287801s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.766016562 (49h36m4.637099607s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:06.426653835 (49h36m6.29773689s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621680284 (49h36m7.492763339s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621710732 (49h36m7.492793777s) [seg_pos={0 0}, upload=43608]: Segment reached optimal threshold of 13 pieces. Dec 7 17:05:07.622110042 (49h36m7.493193097s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection Dec 7 17:05:07.622121895 (49h36m7.49320494s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622169114 (49h36m7.49325216s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: context canceled; context canceled Dec 7 17:05:07.622181257 (49h36m7.493264302s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622414712 (49h36m7.493497757s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: read tcp 188.166.225.170:41952->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 17:05:07.622423679 (49h36m7.493506724s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622449889 (49h36m7.493532924s) [seg_pos={0 0}, upload=43608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 17:05:07.622527457 (49h36m7.493610502s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 17:05:07.622532175 (49h36m7.49361522s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 17:05:07.933366757 (49h36m7.804449802s) [upload=43608]: Done uploading segments... Dec 7 17:05:07.933395712 (49h36m7.804478757s) [upload=43608]: Upload finished. err: <nil> Duration: 5.672416971s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.734530851s trace: https://jaeger.dev.storj.io/trace/2dbb9d950879d488 FAIL (2025-12-07 17:07:09 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 17:04:40.750068351 (49h35m40.621151396s) [upload=43608]: Starting upload Dec 7 17:04:40.750151529 (49h35m40.621234574s) [upload=43608]: Done starting upload Dec 7 17:04:40.750252862 (49h35m40.621335927s) [upload=43608]: Uploading segments... Dec 7 17:04:40.750449186 (49h35m40.621532241s) [upload=43608]: Got next segment. Inline: false Dec 7 17:04:40.750495423 (49h35m40.621578459s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 17:04:40.750502757 (49h35m40.621585802s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 17:04:41.012862673 (49h35m40.883945718s) [seg_pos={0 0}, upload=43608]: Begin upload segment... Dec 7 17:04:41.012921534 (49h35m40.88400458s) [seg_pos={0 0}, upload=43608]: Done begin upload segment. Dec 7 17:04:41.012952173 (49h35m40.884035218s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.014639106 (49h35m40.885722151s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.015553246 (49h35m40.886636291s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016060653 (49h35m40.887143698s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016721020 (49h35m40.887804065s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.018348218 (49h35m40.889431273s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.020177682 (49h35m40.891260727s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.021303415 (49h35m40.89238646s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.023085799 (49h35m40.894168844s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.024447463 (49h35m40.895530498s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.025964030 (49h35m40.897047065s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.027289063 (49h35m40.898372108s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.028891215 (49h35m40.89997425s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.031185423 (49h35m40.902268468s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.032762747 (49h35m40.903845782s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.034246312 (49h35m40.905329347s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:42.710512985 (49h35m42.58159602s) [upload=43608]: Next returned nil segment Dec 7 17:04:42.710525298 (49h35m42.581608343s) [upload=43608]: Waiting for error group managing segments... Dec 7 17:04:43.179511201 (49h35m43.050594246s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:46.586841777 (49h35m46.457924822s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:53.671123584 (49h35m53.542206629s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:55.169666550 (49h35m55.040749595s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:57.066377404 (49h35m56.937460449s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:00.087003610 (49h35m59.958086645s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.688144215 (49h36m2.55922726s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.873597685 (49h36m2.74468073s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:03.747085322 (49h36m3.618168377s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.570204756 (49h36m4.441287801s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.766016562 (49h36m4.637099607s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:06.426653835 (49h36m6.29773689s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621680284 (49h36m7.492763339s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621710732 (49h36m7.492793777s) [seg_pos={0 0}, upload=43608]: Segment reached optimal threshold of 13 pieces. Dec 7 17:05:07.622110042 (49h36m7.493193097s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection Dec 7 17:05:07.622121895 (49h36m7.49320494s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622169114 (49h36m7.49325216s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: context canceled; context canceled Dec 7 17:05:07.622181257 (49h36m7.493264302s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622414712 (49h36m7.493497757s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: read tcp 188.166.225.170:41952->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 17:05:07.622423679 (49h36m7.493506724s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622449889 (49h36m7.493532924s) [seg_pos={0 0}, upload=43608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 17:05:07.622527457 (49h36m7.493610502s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 17:05:07.622532175 (49h36m7.49361522s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 17:05:07.933366757 (49h36m7.804449802s) [upload=43608]: Done uploading segments... Dec 7 17:05:07.933395712 (49h36m7.804478757s) [upload=43608]: Upload finished. err: <nil> Duration: 5.734545519s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.580878091s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 17:07:03 UTC) orbiter warning: context deadline exceeded Duration: 3.581012497s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.430379185s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 17:06:41 UTC) orbiter warning: context deadline exceeded Duration: 3.430409302s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.637680785s trace: https://jaeger.dev.storj.io/trace/2dbb9d950879d488 FAIL (2025-12-07 17:06:39 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 17:04:40.750068351 (49h35m40.621151396s) [upload=43608]: Starting upload Dec 7 17:04:40.750151529 (49h35m40.621234574s) [upload=43608]: Done starting upload Dec 7 17:04:40.750252862 (49h35m40.621335927s) [upload=43608]: Uploading segments... Dec 7 17:04:40.750449186 (49h35m40.621532241s) [upload=43608]: Got next segment. Inline: false Dec 7 17:04:40.750495423 (49h35m40.621578459s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 17:04:40.750502757 (49h35m40.621585802s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 17:04:41.012862673 (49h35m40.883945718s) [seg_pos={0 0}, upload=43608]: Begin upload segment... Dec 7 17:04:41.012921534 (49h35m40.88400458s) [seg_pos={0 0}, upload=43608]: Done begin upload segment. Dec 7 17:04:41.012952173 (49h35m40.884035218s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.014639106 (49h35m40.885722151s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.015553246 (49h35m40.886636291s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016060653 (49h35m40.887143698s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016721020 (49h35m40.887804065s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.018348218 (49h35m40.889431273s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.020177682 (49h35m40.891260727s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.021303415 (49h35m40.89238646s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.023085799 (49h35m40.894168844s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.024447463 (49h35m40.895530498s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.025964030 (49h35m40.897047065s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.027289063 (49h35m40.898372108s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.028891215 (49h35m40.89997425s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.031185423 (49h35m40.902268468s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.032762747 (49h35m40.903845782s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.034246312 (49h35m40.905329347s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:42.710512985 (49h35m42.58159602s) [upload=43608]: Next returned nil segment Dec 7 17:04:42.710525298 (49h35m42.581608343s) [upload=43608]: Waiting for error group managing segments... Dec 7 17:04:43.179511201 (49h35m43.050594246s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:46.586841777 (49h35m46.457924822s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:53.671123584 (49h35m53.542206629s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:55.169666550 (49h35m55.040749595s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:57.066377404 (49h35m56.937460449s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:00.087003610 (49h35m59.958086645s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.688144215 (49h36m2.55922726s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.873597685 (49h36m2.74468073s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:03.747085322 (49h36m3.618168377s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.570204756 (49h36m4.441287801s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.766016562 (49h36m4.637099607s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:06.426653835 (49h36m6.29773689s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621680284 (49h36m7.492763339s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621710732 (49h36m7.492793777s) [seg_pos={0 0}, upload=43608]: Segment reached optimal threshold of 13 pieces. Dec 7 17:05:07.622110042 (49h36m7.493193097s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection Dec 7 17:05:07.622121895 (49h36m7.49320494s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622169114 (49h36m7.49325216s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: context canceled; context canceled Dec 7 17:05:07.622181257 (49h36m7.493264302s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622414712 (49h36m7.493497757s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: read tcp 188.166.225.170:41952->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 17:05:07.622423679 (49h36m7.493506724s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622449889 (49h36m7.493532924s) [seg_pos={0 0}, upload=43608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 17:05:07.622527457 (49h36m7.493610502s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 17:05:07.622532175 (49h36m7.49361522s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 17:05:07.933366757 (49h36m7.804449802s) [upload=43608]: Done uploading segments... Dec 7 17:05:07.933395712 (49h36m7.804478757s) [upload=43608]: Upload finished. err: <nil> Duration: 5.637709289s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X.X.........................................................] duration: 3.488467678s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 17:06:12 UTC) orbiter warning: context deadline exceeded Duration: 3.488578909s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.644965669s trace: https://jaeger.dev.storj.io/trace/2dbb9d950879d488 FAIL (2025-12-07 17:05:43 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 17:04:40.750068351 (49h35m40.621151396s) [upload=43608]: Starting upload Dec 7 17:04:40.750151529 (49h35m40.621234574s) [upload=43608]: Done starting upload Dec 7 17:04:40.750252862 (49h35m40.621335927s) [upload=43608]: Uploading segments... Dec 7 17:04:40.750449186 (49h35m40.621532241s) [upload=43608]: Got next segment. Inline: false Dec 7 17:04:40.750495423 (49h35m40.621578459s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 17:04:40.750502757 (49h35m40.621585802s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 17:04:41.012862673 (49h35m40.883945718s) [seg_pos={0 0}, upload=43608]: Begin upload segment... Dec 7 17:04:41.012921534 (49h35m40.88400458s) [seg_pos={0 0}, upload=43608]: Done begin upload segment. Dec 7 17:04:41.012952173 (49h35m40.884035218s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.014639106 (49h35m40.885722151s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.015553246 (49h35m40.886636291s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016060653 (49h35m40.887143698s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016721020 (49h35m40.887804065s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.018348218 (49h35m40.889431273s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.020177682 (49h35m40.891260727s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.021303415 (49h35m40.89238646s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.023085799 (49h35m40.894168844s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.024447463 (49h35m40.895530498s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.025964030 (49h35m40.897047065s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.027289063 (49h35m40.898372108s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.028891215 (49h35m40.89997425s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.031185423 (49h35m40.902268468s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.032762747 (49h35m40.903845782s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.034246312 (49h35m40.905329347s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:42.710512985 (49h35m42.58159602s) [upload=43608]: Next returned nil segment Dec 7 17:04:42.710525298 (49h35m42.581608343s) [upload=43608]: Waiting for error group managing segments... Dec 7 17:04:43.179511201 (49h35m43.050594246s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:46.586841777 (49h35m46.457924822s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:53.671123584 (49h35m53.542206629s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:55.169666550 (49h35m55.040749595s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:57.066377404 (49h35m56.937460449s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:00.087003610 (49h35m59.958086645s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.688144215 (49h36m2.55922726s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.873597685 (49h36m2.74468073s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:03.747085322 (49h36m3.618168377s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.570204756 (49h36m4.441287801s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.766016562 (49h36m4.637099607s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:06.426653835 (49h36m6.29773689s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621680284 (49h36m7.492763339s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621710732 (49h36m7.492793777s) [seg_pos={0 0}, upload=43608]: Segment reached optimal threshold of 13 pieces. Dec 7 17:05:07.622110042 (49h36m7.493193097s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection Dec 7 17:05:07.622121895 (49h36m7.49320494s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622169114 (49h36m7.49325216s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: context canceled; context canceled Dec 7 17:05:07.622181257 (49h36m7.493264302s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622414712 (49h36m7.493497757s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: read tcp 188.166.225.170:41952->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 17:05:07.622423679 (49h36m7.493506724s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622449889 (49h36m7.493532924s) [seg_pos={0 0}, upload=43608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 17:05:07.622527457 (49h36m7.493610502s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 17:05:07.622532175 (49h36m7.49361522s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 17:05:07.933366757 (49h36m7.804449802s) [upload=43608]: Done uploading segments... Dec 7 17:05:07.933395712 (49h36m7.804478757s) [upload=43608]: Upload finished. err: <nil> Duration: 5.645000465s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [XX.X........................................................] duration: 3.396623196s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 17:05:33 UTC) orbiter warning: context deadline exceeded Duration: 3.396650108s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002719259s trace: https://jaeger.dev.storj.io/trace/2dbb9d950879d488 FAIL (2025-12-07 17:05:10 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 17:04:40.750068351 (49h35m40.621151396s) [upload=43608]: Starting upload Dec 7 17:04:40.750151529 (49h35m40.621234574s) [upload=43608]: Done starting upload Dec 7 17:04:40.750252862 (49h35m40.621335927s) [upload=43608]: Uploading segments... Dec 7 17:04:40.750449186 (49h35m40.621532241s) [upload=43608]: Got next segment. Inline: false Dec 7 17:04:40.750495423 (49h35m40.621578459s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 17:04:40.750502757 (49h35m40.621585802s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 17:04:41.012862673 (49h35m40.883945718s) [seg_pos={0 0}, upload=43608]: Begin upload segment... Dec 7 17:04:41.012921534 (49h35m40.88400458s) [seg_pos={0 0}, upload=43608]: Done begin upload segment. Dec 7 17:04:41.012952173 (49h35m40.884035218s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.014639106 (49h35m40.885722151s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.015553246 (49h35m40.886636291s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016060653 (49h35m40.887143698s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.016721020 (49h35m40.887804065s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.018348218 (49h35m40.889431273s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.020177682 (49h35m40.891260727s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.021303415 (49h35m40.89238646s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.023085799 (49h35m40.894168844s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.024447463 (49h35m40.895530498s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.025964030 (49h35m40.897047065s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.027289063 (49h35m40.898372108s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.028891215 (49h35m40.89997425s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.031185423 (49h35m40.902268468s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.032762747 (49h35m40.903845782s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:41.034246312 (49h35m40.905329347s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Uploading piece... Dec 7 17:04:42.710512985 (49h35m42.58159602s) [upload=43608]: Next returned nil segment Dec 7 17:04:42.710525298 (49h35m42.581608343s) [upload=43608]: Waiting for error group managing segments... Dec 7 17:04:43.179511201 (49h35m43.050594246s) [noise=true , address=77.90.31.15:4699 , piece_id=LDYETSF6H3QRLQP3R5K5ND23WNM6OSCLKT2UNVYDFUWLAXBUAYAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:46.586841777 (49h35m46.457924822s) [noise=true , address=145.79.227.43:4158 , piece_id=UATURKONJQYIHHK5V7JBAM22Q6CAFZVZY2EGLIES2XK4M5QXVFBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:53.671123584 (49h35m53.542206629s) [noise=true , address=145.79.227.28:4725 , piece_id=DEZB2ZETZPFK4JQWARH2BMOBWTHB3LJNGUEI3HCMUIPHBCVRISTQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:55.169666550 (49h35m55.040749595s) [noise=true , address=145.79.227.28:4727 , piece_id=MXQMYHTX43GIYL7QXMJGRKP3CUWVI77UIN55TDNXQ42Z2RR4XCMQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:04:57.066377404 (49h35m56.937460449s) [noise=true , address=145.79.227.40:4050 , piece_id=4LP7RUN433VLXH25GVIBF4KSTJ2AB4JC66BZUTHQRXBMZXBGFGDQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:00.087003610 (49h35m59.958086645s) [noise=true , address=145.79.227.34:4907 , piece_id=GCHR4OE2RJTFC4CXKB2KAPCFRUCCX6EHOTCLKLGBYZROFFDYYZ4Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.688144215 (49h36m2.55922726s) [noise=true , address=145.79.227.29:4941 , piece_id=VOAWSTMHYNYRCHGCSEDLS26KZPXN6DQLUFHJF75VNMIY2JHRS6VA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:02.873597685 (49h36m2.74468073s) [noise=true , address=145.79.227.38:4979 , piece_id=GJBNNFOYPO5Y6ERMDEASIF2KT3X3CVESACNV3X55GM2UEGG4NPEA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:03.747085322 (49h36m3.618168377s) [noise=true , address=145.79.227.44:4195 , piece_id=NXTYROFR2TDL7G6PYNXNIWW66C4OMDSPRDZMYSO5Q22DOPAAMNUA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.570204756 (49h36m4.441287801s) [noise=true , address=145.79.227.23:4546 , piece_id=L5GZIGQNRV42D5JUSSEHZGLZGRWVQHCRJZSXSUNABM2ZK5LNHVBA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:04.766016562 (49h36m4.637099607s) [noise=true , address=145.79.227.33:4869 , piece_id=BZXWNMKNHTJGSHX5RAB2IWPM2AAP6ZXFLQ4726LZDRGAHPZAXD2Q, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:06.426653835 (49h36m6.29773689s) [noise=true , address=145.79.227.39:4013 , piece_id=SHEFK4QV4AOXHBEUKQ5LFDIDBJRRMAAW7I3BFGOIOFVI4UA3LB4A, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621680284 (49h36m7.492763339s) [noise=true , address=145.79.227.42:4121 , piece_id=4T7DFOI2TSKKZDADMZAY57IU2K3AC63O3MPXN26OK2ODK6NYHWOA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: <nil> Dec 7 17:05:07.621710732 (49h36m7.492793777s) [seg_pos={0 0}, upload=43608]: Segment reached optimal threshold of 13 pieces. Dec 7 17:05:07.622110042 (49h36m7.493193097s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53898->145.79.227.27:4690: use of closed network connection Dec 7 17:05:07.622121895 (49h36m7.49320494s) [noise=true , address=145.79.227.27:4690 , piece_id=Q7KXIDF46P6ZIYRZ2URIFK6ZD3JBKIPWRSMXJAPIBPJT4VGHBJNA, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622169114 (49h36m7.49325216s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: context canceled; context canceled Dec 7 17:05:07.622181257 (49h36m7.493264302s) [noise=true , address=145.79.227.43:4157 , piece_id=2BWIWME4ZI3IIDQKCUWCMYE3E3NNW3X4WUYBG6KAHB63ZQAEAZAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622414712 (49h36m7.493497757s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: read tcp 188.166.225.170:41952->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 17:05:07.622423679 (49h36m7.493506724s) [noise=true , address=145.79.227.26:4653 , piece_id=65YZ4VCBYBFYFW7AVODMDRMYMRF4DEMQMMJVJQGSRZEML5QQ7KAQ, seg_pos={0 0}, upload=43608]: Long tail cancellation due to optimal threshold Dec 7 17:05:07.622449889 (49h36m7.493532924s) [seg_pos={0 0}, upload=43608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 17:05:07.622527457 (49h36m7.493610502s) [upload=43608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 17:05:07.622532175 (49h36m7.49361522s) [upload=43608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 17:05:07.933366757 (49h36m7.804449802s) [upload=43608]: Done uploading segments... Dec 7 17:05:07.933395712 (49h36m7.804478757s) [upload=43608]: Upload finished. err: <nil> Duration: 30.002743496s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [XXX.X.......................................................] duration: 3.521860141s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 17:04:33 UTC) orbiter warning: context deadline exceeded Duration: 3.521885621s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.XXX.X......................................................] duration: 3.406488153s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 17:03:23 UTC) orbiter warning: context deadline exceeded Duration: 3.4065131s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [X.XXX.X.....................................................] duration: 4.394429246s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 17:02:17 UTC) orbiter warning: context deadline exceeded Duration: 4.39444717s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.X.XXX.X....................................................] duration: 3.5393835s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 17:01:51 UTC) orbiter warning: context deadline exceeded Duration: 3.539418136s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..X.XXX.X...................................................] duration: 3.49322704s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 17:01:10 UTC) orbiter warning: context deadline exceeded Duration: 3.493255844s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [X..X.XXX.X..................................................] duration: 3.402482503s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 17:00:50 UTC) orbiter warning: context deadline exceeded Duration: 3.402536717s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.X..X.XXX.X.................................................] duration: 3.397093832s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 16:59:39 UTC) orbiter warning: context deadline exceeded Duration: 3.397120963s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..X..X.XXX.X................................................] duration: 3.467471267s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 16:58:49 UTC) orbiter warning: context deadline exceeded Duration: 3.467490933s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...X..X.XXX.X...............................................] duration: 3.521400747s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 16:57:58 UTC) orbiter warning: context deadline exceeded Duration: 3.521425033s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [....X..X.XXX.X..............................................] duration: 3.527967779s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 16:57:42 UTC) orbiter warning: context deadline exceeded Duration: 3.528034194s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.....X..X.XXX.X.............................................] duration: 3.450994469s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 16:56:57 UTC) orbiter warning: context deadline exceeded Duration: 3.451023754s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [......X..X.XXX.X............................................] duration: 3.564557319s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 16:55:55 UTC) orbiter warning: context deadline exceeded Duration: 3.564586443s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.......X..X.XXX.X...........................................] duration: 3.470910072s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 16:54:53 UTC) orbiter warning: context deadline exceeded Duration: 3.470927877s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [........X..X.XXX.X..........................................] duration: 3.435775481s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 16:53:46 UTC) orbiter warning: context deadline exceeded Duration: 3.43580135s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.........X..X.XXX.X.........................................] duration: 3.420665548s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 16:53:11 UTC) orbiter warning: context deadline exceeded Duration: 3.42069756s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..........X..X.XXX.X........................................] duration: 3.507662817s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 16:52:55 UTC) orbiter warning: context deadline exceeded Duration: 3.507716811s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...........X..X.XXX.X.......................................] duration: 3.541473384s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 16:52:03 UTC) orbiter warning: context deadline exceeded Duration: 3.541503693s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [............X..X.XXX.X......................................] duration: 3.544821467s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 16:51:12 UTC) orbiter warning: context deadline exceeded Duration: 3.544851634s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.............X..X.XXX.X.....................................] duration: 3.461741355s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 16:50:49 UTC) orbiter warning: context deadline exceeded Duration: 3.461761134s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..............X..X.XXX.X....................................] duration: 3.456594925s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 16:50:32 UTC) orbiter warning: context deadline exceeded Duration: 3.456623479s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...............X..X.XXX.X...................................] duration: 3.579735903s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 16:50:00 UTC) orbiter warning: context deadline exceeded Duration: 3.579762793s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [................X..X.XXX.X..................................] duration: 4.322021257s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 16:49:05 UTC) orbiter warning: context deadline exceeded Duration: 4.32204427s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.................X..X.XXX.X.................................] duration: 3.493055362s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 16:48:15 UTC) orbiter warning: context deadline exceeded Duration: 3.493082504s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..................X..X.XXX.X................................] duration: 3.420661953s trace: https://jaeger.dev.storj.io/trace/2e1eadccddeca859 FAIL (2025-12-07 16:47:10 UTC) orbiter warning: context deadline exceeded Duration: 3.420710085s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)