Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [....X....X.X..................X...X.....XXX.................] duration: 3.398412815s trace: https://jaeger.dev.storj.io/trace/6d457ad1d7bca31a FAIL (2025-12-08 21:52:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.398594531s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [........X..X......X.....X..X....X.X.........................] duration: 5.768121176s trace: https://jaeger.dev.storj.io/trace/4559eb532569c87c FAIL (2025-12-08 21:52:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.768149592s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.683145098s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:52:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.683175886s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.69764981s trace: https://jaeger.dev.storj.io/trace/855b97200ae87b4 FAIL (2025-12-08 21:52:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ess=45.91.116.13:4177 , piece_id=UDVL2I2NPUJPHYE36BBRZHPWTVJCXTABVOH4AKSP2VWMM74YXEYA, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.680850376 (78h8m7.551933421s) [noise=true , address=77.90.31.13:4659 , piece_id=2VFFOEKI3KXNBHSYR6QEQ3TGVFWXNIXREXGVBCNEEPBCXJ4CEGJQ, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.681583522 (78h8m7.552666557s) [noise=true , address=77.90.31.21:4910 , piece_id=GPUCMH7LE3DYRF7LBE44CBZI5WI52SWO547OLQN3NYTKRX6G6E3A, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.683281806 (78h8m7.554364851s) [noise=true , address=64.49.13.154:4930 , piece_id=GDSMAYDPJ3O22G6TWWWKNKFCUABXZNFKY3S7TRF2T27YQRJVDTBA, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.684363907 (78h8m7.555446952s) [noise=true , address=77.90.31.13:4630 , piece_id=QNSVRMCFKJRFPON6X2BSBZ2JJVVUIVFUHD7RUABAGWXNSPVLWLJA, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.684951405 (78h8m7.556034451s) [noise=true , address=45.91.116.11:4151 , piece_id=SWLFUBM7VIA2W4KZNB4GAC6LMXB32FVEV2ZZJZMRR2Z5CWCZS2YA, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.685755176 (78h8m7.556838211s) [noise=true , address=77.90.31.11:4580 , piece_id=UDT6WNHJCNGWK4RC4UZQCWKTKAVTNRAOQXSJCLGMYJ2PBBJIFVRA, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.686522257 (78h8m7.557605302s) [noise=true , address=5.231.73.2:4001 , piece_id=SQ2CU76U7B52YAPIVWICQEXRKUZ7YH5MZ62YVGA3S2TTHDWICFSA, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.687059629 (78h8m7.558142674s) [noise=true , address=45.91.116.6:4979 , piece_id=DAH4ZK7AKLAEUZQRM6UDFL6RPUMQEPQC7PEWC3T77PDQPMZTYYCQ, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.687708555 (78h8m7.5587916s) [noise=true , address=45.91.116.9:4293 , piece_id=OK66YG5QZ36QF53GMXAPUTDRXCVVI6QNNY7ALPQWIR5D3BEFC5VA, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.688272719 (78h8m7.559355754s) [noise=true , address=77.90.31.19:4876 , piece_id=XVDEFZYYJFOVV57M5QACDKGL6TNZWB5RMMPKRSO3MH7IH66JJLWQ, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.690246739 (78h8m7.561329784s) [noise=true , address=77.90.31.12:4598 , piece_id=GDEFSE4UBVCNGU3G3FQ56HZEAPPKHVVITN6RS7AK6QG7JPYPMB2A, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.691315815 (78h8m7.56239886s) [noise=true , address=145.79.227.44:4205 , piece_id=FIOQM7HZCD5L552DADJYKQUNLM76H7TR5TKSQNPTH3ER5W7JZD7A, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.693506566 (78h8m7.564589611s) [noise=true , address=145.79.227.11:4736 , piece_id=QI3R3YE43LME5BIF74ZZA5MAZYC2JJRWQ3A4AZEHQJIZXKP5EPYA, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.696050741 (78h8m7.567133786s) [noise=true , address=77.90.31.17:4787 , piece_id=A6QRKLXI7XINBSSMX5A36SNBERNC57QQSL3ESDDMDSQND6Z65R6A, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.698540641 (78h8m7.569623686s) [noise=true , address=77.90.31.15:4723 , piece_id=GLC7J3ERGIO3SUSE3BPZ66PJO4NMPSZFX4WJ2EF4EPNNONUWDBBQ, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.701312850 (78h8m7.572395896s) [noise=true , address=145.79.227.15:4886 , piece_id=ADWR7DF2Z6VSVNYHO2VCCMGRHQBGAZ7VL45SSCGHABMZJHDFEU6Q, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.705222034 (78h8m7.576305099s) [noise=true , address=145.79.227.46:4629 , piece_id=PTQDHGJ7GEICGKEWTL555RD5S5CDHRYUSIELMQ726VJ45ZCLH4HQ, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:09.130442999 (78h8m9.001526044s) [upload=68273]: Next returned nil segment Dec 8 21:37:09.130460773 (78h8m9.001543818s) [upload=68273]: Waiting for error group managing segments... Dec 8 21:37:09.539208147 (78h8m9.410291192s) [noise=true , address=5.231.73.2:4001 , piece_id=SQ2CU76U7B52YAPIVWICQEXRKUZ7YH5MZ62YVGA3S2TTHDWICFSA, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.566358075 (78h8m9.43744111s) [noise=true , address=145.79.227.11:4736 , piece_id=QI3R3YE43LME5BIF74ZZA5MAZYC2JJRWQ3A4AZEHQJIZXKP5EPYA, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.568816506 (78h8m9.439899551s) [noise=true , address=145.79.227.46:4629 , piece_id=PTQDHGJ7GEICGKEWTL555RD5S5CDHRYUSIELMQ726VJ45ZCLH4HQ, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.586579460 (78h8m9.457662495s) [noise=true , address=64.49.13.154:4930 , piece_id=GDSMAYDPJ3O22G6TWWWKNKFCUABXZNFKY3S7TRF2T27YQRJVDTBA, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.635764808 (78h8m9.506847843s) [noise=true , address=145.79.227.15:4886 , piece_id=ADWR7DF2Z6VSVNYHO2VCCMGRHQBGAZ7VL45SSCGHABMZJHDFEU6Q, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.650391200 (78h8m9.521474235s) [noise=true , address=77.90.31.19:4876 , piece_id=XVDEFZYYJFOVV57M5QACDKGL6TNZWB5RMMPKRSO3MH7IH66JJLWQ, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.658666919 (78h8m9.529749954s) [noise=true , address=45.91.116.6:4979 , piece_id=DAH4ZK7AKLAEUZQRM6UDFL6RPUMQEPQC7PEWC3T77PDQPMZTYYCQ, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.660460624 (78h8m9.531543659s) [noise=true , address=77.90.31.13:4630 , piece_id=QNSVRMCFKJRFPON6X2BSBZ2JJVVUIVFUHD7RUABAGWXNSPVLWLJA, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.680325200 (78h8m9.551408245s) [noise=true , address=77.90.31.13:4659 , piece_id=2VFFOEKI3KXNBHSYR6QEQ3TGVFWXNIXREXGVBCNEEPBCXJ4CEGJQ, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.700227848 (78h8m9.571310893s) [noise=true , address=77.90.31.17:4787 , piece_id=A6QRKLXI7XINBSSMX5A36SNBERNC57QQSL3ESDDMDSQND6Z65R6A, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.713665697 (78h8m9.584748742s) [noise=true , address=45.91.116.11:4151 , piece_id=SWLFUBM7VIA2W4KZNB4GAC6LMXB32FVEV2ZZJZMRR2Z5CWCZS2YA, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.720646650 (78h8m9.591729695s) [noise=true , address=77.90.31.12:4598 , piece_id=GDEFSE4UBVCNGU3G3FQ56HZEAPPKHVVITN6RS7AK6QG7JPYPMB2A, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.727453051 (78h8m9.598536096s) [noise=true , address=77.90.31.21:4910 , piece_id=GPUCMH7LE3DYRF7LBE44CBZI5WI52SWO547OLQN3NYTKRX6G6E3A, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.757420364 (78h8m9.628503399s) [noise=true , address=77.90.31.15:4723 , piece_id=GLC7J3ERGIO3SUSE3BPZ66PJO4NMPSZFX4WJ2EF4EPNNONUWDBBQ, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.757446764 (78h8m9.628529799s) [seg_pos={0 0}, upload=68273]: Segment reached optimal threshold of 14 pieces. Dec 8 21:37:09.757706399 (78h8m9.628789444s) [noise=true , address=77.90.31.11:4580 , piece_id=UDT6WNHJCNGWK4RC4UZQCWKTKAVTNRAOQXSJCLGMYJ2PBBJIFVRA, seg_pos={0 0}, upload=68273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D): protocol: expected piece hash; context canceled; EOF Dec 8 21:37:09.757716188 (78h8m9.628799223s) [noise=true , address=77.90.31.11:4580 , piece_id=UDT6WNHJCNGWK4RC4UZQCWKTKAVTNRAOQXSJCLGMYJ2PBBJIFVRA, seg_pos={0 0}, upload=68273]: Long tail cancellation due to optimal threshold Dec 8 21:37:09.758299839 (78h8m9.629382884s) [noise=true , address=45.91.116.13:4177 , piece_id=UDVL2I2NPUJPHYE36BBRZHPWTVJCXTABVOH4AKSP2VWMM74YXEYA, seg_pos={0 0}, upload=68273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwaFojWAo54xD34ZA7HACYmJf811KFqN9biJJoag5oxE9fJqx): protocol: expected piece hash; context canceled; EOF Dec 8 21:37:09.758318384 (78h8m9.629401429s) [noise=true , address=45.91.116.13:4177 , piece_id=UDVL2I2NPUJPHYE36BBRZHPWTVJCXTABVOH4AKSP2VWMM74YXEYA, seg_pos={0 0}, upload=68273]: Long tail cancellation due to optimal threshold Dec 8 21:37:09.758479150 (78h8m9.629562196s) [noise=true , address=45.91.116.9:4293 , piece_id=OK66YG5QZ36QF53GMXAPUTDRXCVVI6QNNY7ALPQWIR5D3BEFC5VA, seg_pos={0 0}, upload=68273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG): protocol: expected piece hash; context canceled; EOF Dec 8 21:37:09.758498306 (78h8m9.629581351s) [noise=true , address=45.91.116.9:4293 , piece_id=OK66YG5QZ36QF53GMXAPUTDRXCVVI6QNNY7ALPQWIR5D3BEFC5VA, seg_pos={0 0}, upload=68273]: Long tail cancellation due to optimal threshold Dec 8 21:37:09.758624106 (78h8m9.629707151s) [noise=true , address=145.79.227.44:4205 , piece_id=FIOQM7HZCD5L552DADJYKQUNLM76H7TR5TKSQNPTH3ER5W7JZD7A, seg_pos={0 0}, upload=68273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UafxBVo9nuRVBcQmGE4gBP2pnXQ5YUrwHZdwPCeSBSBWCZKpQ): protocol: expected piece hash; context canceled; EOF Dec 8 21:37:09.758635287 (78h8m9.629718332s) [noise=true , address=145.79.227.44:4205 , piece_id=FIOQM7HZCD5L552DADJYKQUNLM76H7TR5TKSQNPTH3ER5W7JZD7A, seg_pos={0 0}, upload=68273]: Long tail cancellation due to optimal threshold Dec 8 21:37:09.758667849 (78h8m9.629750884s) [seg_pos={0 0}, upload=68273]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 21:37:09.758866318 (78h8m9.629949363s) [upload=68273]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:37:09.758874373 (78h8m9.629957418s) [upload=68273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:51:34.220671681 (78h22m34.091754716s) [upload=68273]: Done uploading segments... Dec 8 21:51:34.220721916 (78h22m34.091804951s) [upload=68273]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.697679005s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.....X....X.X..................X...X.....XXX................] duration: 3.530400846s trace: https://jaeger.dev.storj.io/trace/6d457ad1d7bca31a FAIL (2025-12-08 21:52:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.53042962s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.........X............................................] duration: 5.714815725s trace: https://jaeger.dev.storj.io/trace/5fd4c7e9abd66483 FAIL (2025-12-08 21:52:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.714846614s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.701871195s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:51:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.701899579s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.002824673s trace: https://jaeger.dev.storj.io/trace/1ea7632defe62c27 FAIL (2025-12-08 21:51:56 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.002853027s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.........X..X......X.....X..X....X.X........................] duration: 5.825119261s trace: https://jaeger.dev.storj.io/trace/4559eb532569c87c FAIL (2025-12-08 21:51:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.825147705s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.727302863s trace: https://jaeger.dev.storj.io/trace/855b97200ae87b4 FAIL (2025-12-08 21:51:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ess=45.91.116.13:4177 , piece_id=UDVL2I2NPUJPHYE36BBRZHPWTVJCXTABVOH4AKSP2VWMM74YXEYA, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.680850376 (78h8m7.551933421s) [noise=true , address=77.90.31.13:4659 , piece_id=2VFFOEKI3KXNBHSYR6QEQ3TGVFWXNIXREXGVBCNEEPBCXJ4CEGJQ, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.681583522 (78h8m7.552666557s) [noise=true , address=77.90.31.21:4910 , piece_id=GPUCMH7LE3DYRF7LBE44CBZI5WI52SWO547OLQN3NYTKRX6G6E3A, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.683281806 (78h8m7.554364851s) [noise=true , address=64.49.13.154:4930 , piece_id=GDSMAYDPJ3O22G6TWWWKNKFCUABXZNFKY3S7TRF2T27YQRJVDTBA, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.684363907 (78h8m7.555446952s) [noise=true , address=77.90.31.13:4630 , piece_id=QNSVRMCFKJRFPON6X2BSBZ2JJVVUIVFUHD7RUABAGWXNSPVLWLJA, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.684951405 (78h8m7.556034451s) [noise=true , address=45.91.116.11:4151 , piece_id=SWLFUBM7VIA2W4KZNB4GAC6LMXB32FVEV2ZZJZMRR2Z5CWCZS2YA, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.685755176 (78h8m7.556838211s) [noise=true , address=77.90.31.11:4580 , piece_id=UDT6WNHJCNGWK4RC4UZQCWKTKAVTNRAOQXSJCLGMYJ2PBBJIFVRA, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.686522257 (78h8m7.557605302s) [noise=true , address=5.231.73.2:4001 , piece_id=SQ2CU76U7B52YAPIVWICQEXRKUZ7YH5MZ62YVGA3S2TTHDWICFSA, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.687059629 (78h8m7.558142674s) [noise=true , address=45.91.116.6:4979 , piece_id=DAH4ZK7AKLAEUZQRM6UDFL6RPUMQEPQC7PEWC3T77PDQPMZTYYCQ, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.687708555 (78h8m7.5587916s) [noise=true , address=45.91.116.9:4293 , piece_id=OK66YG5QZ36QF53GMXAPUTDRXCVVI6QNNY7ALPQWIR5D3BEFC5VA, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.688272719 (78h8m7.559355754s) [noise=true , address=77.90.31.19:4876 , piece_id=XVDEFZYYJFOVV57M5QACDKGL6TNZWB5RMMPKRSO3MH7IH66JJLWQ, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.690246739 (78h8m7.561329784s) [noise=true , address=77.90.31.12:4598 , piece_id=GDEFSE4UBVCNGU3G3FQ56HZEAPPKHVVITN6RS7AK6QG7JPYPMB2A, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.691315815 (78h8m7.56239886s) [noise=true , address=145.79.227.44:4205 , piece_id=FIOQM7HZCD5L552DADJYKQUNLM76H7TR5TKSQNPTH3ER5W7JZD7A, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.693506566 (78h8m7.564589611s) [noise=true , address=145.79.227.11:4736 , piece_id=QI3R3YE43LME5BIF74ZZA5MAZYC2JJRWQ3A4AZEHQJIZXKP5EPYA, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.696050741 (78h8m7.567133786s) [noise=true , address=77.90.31.17:4787 , piece_id=A6QRKLXI7XINBSSMX5A36SNBERNC57QQSL3ESDDMDSQND6Z65R6A, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.698540641 (78h8m7.569623686s) [noise=true , address=77.90.31.15:4723 , piece_id=GLC7J3ERGIO3SUSE3BPZ66PJO4NMPSZFX4WJ2EF4EPNNONUWDBBQ, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.701312850 (78h8m7.572395896s) [noise=true , address=145.79.227.15:4886 , piece_id=ADWR7DF2Z6VSVNYHO2VCCMGRHQBGAZ7VL45SSCGHABMZJHDFEU6Q, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.705222034 (78h8m7.576305099s) [noise=true , address=145.79.227.46:4629 , piece_id=PTQDHGJ7GEICGKEWTL555RD5S5CDHRYUSIELMQ726VJ45ZCLH4HQ, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:09.130442999 (78h8m9.001526044s) [upload=68273]: Next returned nil segment Dec 8 21:37:09.130460773 (78h8m9.001543818s) [upload=68273]: Waiting for error group managing segments... Dec 8 21:37:09.539208147 (78h8m9.410291192s) [noise=true , address=5.231.73.2:4001 , piece_id=SQ2CU76U7B52YAPIVWICQEXRKUZ7YH5MZ62YVGA3S2TTHDWICFSA, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.566358075 (78h8m9.43744111s) [noise=true , address=145.79.227.11:4736 , piece_id=QI3R3YE43LME5BIF74ZZA5MAZYC2JJRWQ3A4AZEHQJIZXKP5EPYA, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.568816506 (78h8m9.439899551s) [noise=true , address=145.79.227.46:4629 , piece_id=PTQDHGJ7GEICGKEWTL555RD5S5CDHRYUSIELMQ726VJ45ZCLH4HQ, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.586579460 (78h8m9.457662495s) [noise=true , address=64.49.13.154:4930 , piece_id=GDSMAYDPJ3O22G6TWWWKNKFCUABXZNFKY3S7TRF2T27YQRJVDTBA, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.635764808 (78h8m9.506847843s) [noise=true , address=145.79.227.15:4886 , piece_id=ADWR7DF2Z6VSVNYHO2VCCMGRHQBGAZ7VL45SSCGHABMZJHDFEU6Q, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.650391200 (78h8m9.521474235s) [noise=true , address=77.90.31.19:4876 , piece_id=XVDEFZYYJFOVV57M5QACDKGL6TNZWB5RMMPKRSO3MH7IH66JJLWQ, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.658666919 (78h8m9.529749954s) [noise=true , address=45.91.116.6:4979 , piece_id=DAH4ZK7AKLAEUZQRM6UDFL6RPUMQEPQC7PEWC3T77PDQPMZTYYCQ, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.660460624 (78h8m9.531543659s) [noise=true , address=77.90.31.13:4630 , piece_id=QNSVRMCFKJRFPON6X2BSBZ2JJVVUIVFUHD7RUABAGWXNSPVLWLJA, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.680325200 (78h8m9.551408245s) [noise=true , address=77.90.31.13:4659 , piece_id=2VFFOEKI3KXNBHSYR6QEQ3TGVFWXNIXREXGVBCNEEPBCXJ4CEGJQ, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.700227848 (78h8m9.571310893s) [noise=true , address=77.90.31.17:4787 , piece_id=A6QRKLXI7XINBSSMX5A36SNBERNC57QQSL3ESDDMDSQND6Z65R6A, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.713665697 (78h8m9.584748742s) [noise=true , address=45.91.116.11:4151 , piece_id=SWLFUBM7VIA2W4KZNB4GAC6LMXB32FVEV2ZZJZMRR2Z5CWCZS2YA, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.720646650 (78h8m9.591729695s) [noise=true , address=77.90.31.12:4598 , piece_id=GDEFSE4UBVCNGU3G3FQ56HZEAPPKHVVITN6RS7AK6QG7JPYPMB2A, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.727453051 (78h8m9.598536096s) [noise=true , address=77.90.31.21:4910 , piece_id=GPUCMH7LE3DYRF7LBE44CBZI5WI52SWO547OLQN3NYTKRX6G6E3A, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.757420364 (78h8m9.628503399s) [noise=true , address=77.90.31.15:4723 , piece_id=GLC7J3ERGIO3SUSE3BPZ66PJO4NMPSZFX4WJ2EF4EPNNONUWDBBQ, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.757446764 (78h8m9.628529799s) [seg_pos={0 0}, upload=68273]: Segment reached optimal threshold of 14 pieces. Dec 8 21:37:09.757706399 (78h8m9.628789444s) [noise=true , address=77.90.31.11:4580 , piece_id=UDT6WNHJCNGWK4RC4UZQCWKTKAVTNRAOQXSJCLGMYJ2PBBJIFVRA, seg_pos={0 0}, upload=68273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D): protocol: expected piece hash; context canceled; EOF Dec 8 21:37:09.757716188 (78h8m9.628799223s) [noise=true , address=77.90.31.11:4580 , piece_id=UDT6WNHJCNGWK4RC4UZQCWKTKAVTNRAOQXSJCLGMYJ2PBBJIFVRA, seg_pos={0 0}, upload=68273]: Long tail cancellation due to optimal threshold Dec 8 21:37:09.758299839 (78h8m9.629382884s) [noise=true , address=45.91.116.13:4177 , piece_id=UDVL2I2NPUJPHYE36BBRZHPWTVJCXTABVOH4AKSP2VWMM74YXEYA, seg_pos={0 0}, upload=68273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwaFojWAo54xD34ZA7HACYmJf811KFqN9biJJoag5oxE9fJqx): protocol: expected piece hash; context canceled; EOF Dec 8 21:37:09.758318384 (78h8m9.629401429s) [noise=true , address=45.91.116.13:4177 , piece_id=UDVL2I2NPUJPHYE36BBRZHPWTVJCXTABVOH4AKSP2VWMM74YXEYA, seg_pos={0 0}, upload=68273]: Long tail cancellation due to optimal threshold Dec 8 21:37:09.758479150 (78h8m9.629562196s) [noise=true , address=45.91.116.9:4293 , piece_id=OK66YG5QZ36QF53GMXAPUTDRXCVVI6QNNY7ALPQWIR5D3BEFC5VA, seg_pos={0 0}, upload=68273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG): protocol: expected piece hash; context canceled; EOF Dec 8 21:37:09.758498306 (78h8m9.629581351s) [noise=true , address=45.91.116.9:4293 , piece_id=OK66YG5QZ36QF53GMXAPUTDRXCVVI6QNNY7ALPQWIR5D3BEFC5VA, seg_pos={0 0}, upload=68273]: Long tail cancellation due to optimal threshold Dec 8 21:37:09.758624106 (78h8m9.629707151s) [noise=true , address=145.79.227.44:4205 , piece_id=FIOQM7HZCD5L552DADJYKQUNLM76H7TR5TKSQNPTH3ER5W7JZD7A, seg_pos={0 0}, upload=68273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UafxBVo9nuRVBcQmGE4gBP2pnXQ5YUrwHZdwPCeSBSBWCZKpQ): protocol: expected piece hash; context canceled; EOF Dec 8 21:37:09.758635287 (78h8m9.629718332s) [noise=true , address=145.79.227.44:4205 , piece_id=FIOQM7HZCD5L552DADJYKQUNLM76H7TR5TKSQNPTH3ER5W7JZD7A, seg_pos={0 0}, upload=68273]: Long tail cancellation due to optimal threshold Dec 8 21:37:09.758667849 (78h8m9.629750884s) [seg_pos={0 0}, upload=68273]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 21:37:09.758866318 (78h8m9.629949363s) [upload=68273]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:37:09.758874373 (78h8m9.629957418s) [upload=68273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:51:34.220671681 (78h22m34.091754716s) [upload=68273]: Done uploading segments... Dec 8 21:51:34.220721916 (78h22m34.091804951s) [upload=68273]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.727317009s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.955895783s trace: https://jaeger.dev.storj.io/trace/1ea7632defe62c27 FAIL (2025-12-08 21:51:37 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.955919598s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 14m28.492372828s trace: https://jaeger.dev.storj.io/trace/855b97200ae87b4 FAIL (2025-12-08 21:51:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ess=45.91.116.13:4177 , piece_id=UDVL2I2NPUJPHYE36BBRZHPWTVJCXTABVOH4AKSP2VWMM74YXEYA, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.680850376 (78h8m7.551933421s) [noise=true , address=77.90.31.13:4659 , piece_id=2VFFOEKI3KXNBHSYR6QEQ3TGVFWXNIXREXGVBCNEEPBCXJ4CEGJQ, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.681583522 (78h8m7.552666557s) [noise=true , address=77.90.31.21:4910 , piece_id=GPUCMH7LE3DYRF7LBE44CBZI5WI52SWO547OLQN3NYTKRX6G6E3A, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.683281806 (78h8m7.554364851s) [noise=true , address=64.49.13.154:4930 , piece_id=GDSMAYDPJ3O22G6TWWWKNKFCUABXZNFKY3S7TRF2T27YQRJVDTBA, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.684363907 (78h8m7.555446952s) [noise=true , address=77.90.31.13:4630 , piece_id=QNSVRMCFKJRFPON6X2BSBZ2JJVVUIVFUHD7RUABAGWXNSPVLWLJA, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.684951405 (78h8m7.556034451s) [noise=true , address=45.91.116.11:4151 , piece_id=SWLFUBM7VIA2W4KZNB4GAC6LMXB32FVEV2ZZJZMRR2Z5CWCZS2YA, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.685755176 (78h8m7.556838211s) [noise=true , address=77.90.31.11:4580 , piece_id=UDT6WNHJCNGWK4RC4UZQCWKTKAVTNRAOQXSJCLGMYJ2PBBJIFVRA, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.686522257 (78h8m7.557605302s) [noise=true , address=5.231.73.2:4001 , piece_id=SQ2CU76U7B52YAPIVWICQEXRKUZ7YH5MZ62YVGA3S2TTHDWICFSA, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.687059629 (78h8m7.558142674s) [noise=true , address=45.91.116.6:4979 , piece_id=DAH4ZK7AKLAEUZQRM6UDFL6RPUMQEPQC7PEWC3T77PDQPMZTYYCQ, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.687708555 (78h8m7.5587916s) [noise=true , address=45.91.116.9:4293 , piece_id=OK66YG5QZ36QF53GMXAPUTDRXCVVI6QNNY7ALPQWIR5D3BEFC5VA, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.688272719 (78h8m7.559355754s) [noise=true , address=77.90.31.19:4876 , piece_id=XVDEFZYYJFOVV57M5QACDKGL6TNZWB5RMMPKRSO3MH7IH66JJLWQ, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.690246739 (78h8m7.561329784s) [noise=true , address=77.90.31.12:4598 , piece_id=GDEFSE4UBVCNGU3G3FQ56HZEAPPKHVVITN6RS7AK6QG7JPYPMB2A, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.691315815 (78h8m7.56239886s) [noise=true , address=145.79.227.44:4205 , piece_id=FIOQM7HZCD5L552DADJYKQUNLM76H7TR5TKSQNPTH3ER5W7JZD7A, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.693506566 (78h8m7.564589611s) [noise=true , address=145.79.227.11:4736 , piece_id=QI3R3YE43LME5BIF74ZZA5MAZYC2JJRWQ3A4AZEHQJIZXKP5EPYA, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.696050741 (78h8m7.567133786s) [noise=true , address=77.90.31.17:4787 , piece_id=A6QRKLXI7XINBSSMX5A36SNBERNC57QQSL3ESDDMDSQND6Z65R6A, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.698540641 (78h8m7.569623686s) [noise=true , address=77.90.31.15:4723 , piece_id=GLC7J3ERGIO3SUSE3BPZ66PJO4NMPSZFX4WJ2EF4EPNNONUWDBBQ, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.701312850 (78h8m7.572395896s) [noise=true , address=145.79.227.15:4886 , piece_id=ADWR7DF2Z6VSVNYHO2VCCMGRHQBGAZ7VL45SSCGHABMZJHDFEU6Q, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:07.705222034 (78h8m7.576305099s) [noise=true , address=145.79.227.46:4629 , piece_id=PTQDHGJ7GEICGKEWTL555RD5S5CDHRYUSIELMQ726VJ45ZCLH4HQ, seg_pos={0 0}, upload=68273]: Uploading piece... Dec 8 21:37:09.130442999 (78h8m9.001526044s) [upload=68273]: Next returned nil segment Dec 8 21:37:09.130460773 (78h8m9.001543818s) [upload=68273]: Waiting for error group managing segments... Dec 8 21:37:09.539208147 (78h8m9.410291192s) [noise=true , address=5.231.73.2:4001 , piece_id=SQ2CU76U7B52YAPIVWICQEXRKUZ7YH5MZ62YVGA3S2TTHDWICFSA, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.566358075 (78h8m9.43744111s) [noise=true , address=145.79.227.11:4736 , piece_id=QI3R3YE43LME5BIF74ZZA5MAZYC2JJRWQ3A4AZEHQJIZXKP5EPYA, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.568816506 (78h8m9.439899551s) [noise=true , address=145.79.227.46:4629 , piece_id=PTQDHGJ7GEICGKEWTL555RD5S5CDHRYUSIELMQ726VJ45ZCLH4HQ, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.586579460 (78h8m9.457662495s) [noise=true , address=64.49.13.154:4930 , piece_id=GDSMAYDPJ3O22G6TWWWKNKFCUABXZNFKY3S7TRF2T27YQRJVDTBA, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.635764808 (78h8m9.506847843s) [noise=true , address=145.79.227.15:4886 , piece_id=ADWR7DF2Z6VSVNYHO2VCCMGRHQBGAZ7VL45SSCGHABMZJHDFEU6Q, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.650391200 (78h8m9.521474235s) [noise=true , address=77.90.31.19:4876 , piece_id=XVDEFZYYJFOVV57M5QACDKGL6TNZWB5RMMPKRSO3MH7IH66JJLWQ, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.658666919 (78h8m9.529749954s) [noise=true , address=45.91.116.6:4979 , piece_id=DAH4ZK7AKLAEUZQRM6UDFL6RPUMQEPQC7PEWC3T77PDQPMZTYYCQ, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.660460624 (78h8m9.531543659s) [noise=true , address=77.90.31.13:4630 , piece_id=QNSVRMCFKJRFPON6X2BSBZ2JJVVUIVFUHD7RUABAGWXNSPVLWLJA, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.680325200 (78h8m9.551408245s) [noise=true , address=77.90.31.13:4659 , piece_id=2VFFOEKI3KXNBHSYR6QEQ3TGVFWXNIXREXGVBCNEEPBCXJ4CEGJQ, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.700227848 (78h8m9.571310893s) [noise=true , address=77.90.31.17:4787 , piece_id=A6QRKLXI7XINBSSMX5A36SNBERNC57QQSL3ESDDMDSQND6Z65R6A, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.713665697 (78h8m9.584748742s) [noise=true , address=45.91.116.11:4151 , piece_id=SWLFUBM7VIA2W4KZNB4GAC6LMXB32FVEV2ZZJZMRR2Z5CWCZS2YA, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.720646650 (78h8m9.591729695s) [noise=true , address=77.90.31.12:4598 , piece_id=GDEFSE4UBVCNGU3G3FQ56HZEAPPKHVVITN6RS7AK6QG7JPYPMB2A, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.727453051 (78h8m9.598536096s) [noise=true , address=77.90.31.21:4910 , piece_id=GPUCMH7LE3DYRF7LBE44CBZI5WI52SWO547OLQN3NYTKRX6G6E3A, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.757420364 (78h8m9.628503399s) [noise=true , address=77.90.31.15:4723 , piece_id=GLC7J3ERGIO3SUSE3BPZ66PJO4NMPSZFX4WJ2EF4EPNNONUWDBBQ, seg_pos={0 0}, upload=68273]: Done uploading piece. err: <nil> Dec 8 21:37:09.757446764 (78h8m9.628529799s) [seg_pos={0 0}, upload=68273]: Segment reached optimal threshold of 14 pieces. Dec 8 21:37:09.757706399 (78h8m9.628789444s) [noise=true , address=77.90.31.11:4580 , piece_id=UDT6WNHJCNGWK4RC4UZQCWKTKAVTNRAOQXSJCLGMYJ2PBBJIFVRA, seg_pos={0 0}, upload=68273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D): protocol: expected piece hash; context canceled; EOF Dec 8 21:37:09.757716188 (78h8m9.628799223s) [noise=true , address=77.90.31.11:4580 , piece_id=UDT6WNHJCNGWK4RC4UZQCWKTKAVTNRAOQXSJCLGMYJ2PBBJIFVRA, seg_pos={0 0}, upload=68273]: Long tail cancellation due to optimal threshold Dec 8 21:37:09.758299839 (78h8m9.629382884s) [noise=true , address=45.91.116.13:4177 , piece_id=UDVL2I2NPUJPHYE36BBRZHPWTVJCXTABVOH4AKSP2VWMM74YXEYA, seg_pos={0 0}, upload=68273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwaFojWAo54xD34ZA7HACYmJf811KFqN9biJJoag5oxE9fJqx): protocol: expected piece hash; context canceled; EOF Dec 8 21:37:09.758318384 (78h8m9.629401429s) [noise=true , address=45.91.116.13:4177 , piece_id=UDVL2I2NPUJPHYE36BBRZHPWTVJCXTABVOH4AKSP2VWMM74YXEYA, seg_pos={0 0}, upload=68273]: Long tail cancellation due to optimal threshold Dec 8 21:37:09.758479150 (78h8m9.629562196s) [noise=true , address=45.91.116.9:4293 , piece_id=OK66YG5QZ36QF53GMXAPUTDRXCVVI6QNNY7ALPQWIR5D3BEFC5VA, seg_pos={0 0}, upload=68273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG): protocol: expected piece hash; context canceled; EOF Dec 8 21:37:09.758498306 (78h8m9.629581351s) [noise=true , address=45.91.116.9:4293 , piece_id=OK66YG5QZ36QF53GMXAPUTDRXCVVI6QNNY7ALPQWIR5D3BEFC5VA, seg_pos={0 0}, upload=68273]: Long tail cancellation due to optimal threshold Dec 8 21:37:09.758624106 (78h8m9.629707151s) [noise=true , address=145.79.227.44:4205 , piece_id=FIOQM7HZCD5L552DADJYKQUNLM76H7TR5TKSQNPTH3ER5W7JZD7A, seg_pos={0 0}, upload=68273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UafxBVo9nuRVBcQmGE4gBP2pnXQ5YUrwHZdwPCeSBSBWCZKpQ): protocol: expected piece hash; context canceled; EOF Dec 8 21:37:09.758635287 (78h8m9.629718332s) [noise=true , address=145.79.227.44:4205 , piece_id=FIOQM7HZCD5L552DADJYKQUNLM76H7TR5TKSQNPTH3ER5W7JZD7A, seg_pos={0 0}, upload=68273]: Long tail cancellation due to optimal threshold Dec 8 21:37:09.758667849 (78h8m9.629750884s) [seg_pos={0 0}, upload=68273]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 21:37:09.758866318 (78h8m9.629949363s) [upload=68273]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:37:09.758874373 (78h8m9.629957418s) [upload=68273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:51:34.220671681 (78h22m34.091754716s) [upload=68273]: Done uploading segments... Dec 8 21:51:34.220721916 (78h22m34.091804951s) [upload=68273]: Upload finished. err: metaclient: context deadline exceeded Duration: 14m28.492398547s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.........X...........................................] duration: 5.86506323s trace: https://jaeger.dev.storj.io/trace/5fd4c7e9abd66483 FAIL (2025-12-08 21:51:31 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.865088499s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [......X....X.X..................X...X.....XXX...............] duration: 3.534311068s trace: https://jaeger.dev.storj.io/trace/6d457ad1d7bca31a FAIL (2025-12-08 21:51:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.534335304s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.........X..........................................] duration: 5.795439466s trace: https://jaeger.dev.storj.io/trace/5fd4c7e9abd66483 FAIL (2025-12-08 21:51:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.795487054s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..........X..X......X.....X..X....X.X.......................] duration: 5.821235243s trace: https://jaeger.dev.storj.io/trace/4559eb532569c87c FAIL (2025-12-08 21:51:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.821261523s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 1.975445116s trace: https://jaeger.dev.storj.io/trace/1ea7632defe62c27 FAIL (2025-12-08 21:50:52 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.975476216s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.666994023s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:50:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.66701349s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [...........X..X......X.....X..X....X.X......................] duration: 5.890229561s trace: https://jaeger.dev.storj.io/trace/4559eb532569c87c FAIL (2025-12-08 21:50:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.890259407s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.......X....X.X..................X...X.....XXX..............] duration: 3.520410889s trace: https://jaeger.dev.storj.io/trace/6d457ad1d7bca31a FAIL (2025-12-08 21:50:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.520433081s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.......X.........X.........................................] duration: 6.100143965s trace: https://jaeger.dev.storj.io/trace/5fd4c7e9abd66483 FAIL (2025-12-08 21:50:05 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.10017336s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.029350993s trace: https://jaeger.dev.storj.io/trace/1ea7632defe62c27 FAIL (2025-12-08 21:50:05 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.029371803s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.760600828s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:49:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.760630615s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [........X....X.X..................X...X.....XXX.............] duration: 3.439166115s trace: https://jaeger.dev.storj.io/trace/6d457ad1d7bca31a FAIL (2025-12-08 21:49:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.439191965s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.017278365s trace: https://jaeger.dev.storj.io/trace/1ea7632defe62c27 FAIL (2025-12-08 21:49:46 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.017309224s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [............X..X......X.....X..X....X.X.....................] duration: 5.747367709s trace: https://jaeger.dev.storj.io/trace/4559eb532569c87c FAIL (2025-12-08 21:49:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.747392667s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.694523695s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:49:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.694554393s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.......X.........X........................................] duration: 5.750320933s trace: https://jaeger.dev.storj.io/trace/5fd4c7e9abd66483 FAIL (2025-12-08 21:49:14 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.750368022s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 1.990893623s trace: https://jaeger.dev.storj.io/trace/1ea7632defe62c27 FAIL (2025-12-08 21:48:53 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.990924653s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.........X....X.X..................X...X.....XXX............] duration: 3.475586047s trace: https://jaeger.dev.storj.io/trace/6d457ad1d7bca31a FAIL (2025-12-08 21:48:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.475612807s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.............X..X......X.....X..X....X.X....................] duration: 5.737249102s trace: https://jaeger.dev.storj.io/trace/4559eb532569c87c FAIL (2025-12-08 21:48:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.737312632s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..........X....X.X..................X...X.....XXX...........] duration: 3.556245504s trace: https://jaeger.dev.storj.io/trace/6d457ad1d7bca31a FAIL (2025-12-08 21:48:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.556278468s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.72360578s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:48:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.723635346s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.......X.........X.......................................] duration: 5.837404504s trace: https://jaeger.dev.storj.io/trace/5fd4c7e9abd66483 FAIL (2025-12-08 21:48:13 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.837433159s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.056214004s trace: https://jaeger.dev.storj.io/trace/1ea7632defe62c27 FAIL (2025-12-08 21:48:13 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.056243121s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.......X.........X......................................] duration: 5.798844221s trace: https://jaeger.dev.storj.io/trace/5fd4c7e9abd66483 FAIL (2025-12-08 21:47:48 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.798876041s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...........X....X.X..................X...X.....XXX..........] duration: 3.397252909s trace: https://jaeger.dev.storj.io/trace/6d457ad1d7bca31a FAIL (2025-12-08 21:47:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.397281384s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 1.999372902s trace: https://jaeger.dev.storj.io/trace/1ea7632defe62c27 FAIL (2025-12-08 21:47:28 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.999399433s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..............X..X......X.....X..X....X.X...................] duration: 6.723739856s trace: https://jaeger.dev.storj.io/trace/4559eb532569c87c FAIL (2025-12-08 21:47:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.723769393s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.726934089s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:47:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.726962083s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [............X....X.X..................X...X.....XXX.........] duration: 3.686306686s trace: https://jaeger.dev.storj.io/trace/6d457ad1d7bca31a FAIL (2025-12-08 21:47:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.686339227s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.......X.........X.....................................] duration: 5.858103278s trace: https://jaeger.dev.storj.io/trace/5fd4c7e9abd66483 FAIL (2025-12-08 21:47:01 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.8581309s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.050722999s trace: https://jaeger.dev.storj.io/trace/1ea7632defe62c27 FAIL (2025-12-08 21:46:55 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.050750561s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [...............X..X......X.....X..X....X.X..................] duration: 5.716115581s trace: https://jaeger.dev.storj.io/trace/4559eb532569c87c FAIL (2025-12-08 21:46:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.716190082s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 1.986254117s trace: https://jaeger.dev.storj.io/trace/1ea7632defe62c27 FAIL (2025-12-08 21:46:29 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.986283775s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.......X.........X....................................] duration: 5.893106912s trace: https://jaeger.dev.storj.io/trace/5fd4c7e9abd66483 FAIL (2025-12-08 21:46:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.89325364s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.............X....X.X..................X...X.....XXX........] duration: 3.439126722s trace: https://jaeger.dev.storj.io/trace/6d457ad1d7bca31a FAIL (2025-12-08 21:46:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.439172721s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.744312762s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:46:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.744341968s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.026689805s trace: https://jaeger.dev.storj.io/trace/1ea7632defe62c27 FAIL (2025-12-08 21:45:46 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.026719563s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..............X....X.X..................X...X.....XXX.......] duration: 3.476309013s trace: https://jaeger.dev.storj.io/trace/6d457ad1d7bca31a FAIL (2025-12-08 21:45:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.476340593s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)