Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.......X......................X....X.X..................X...] duration: 3.507927287s trace: https://jaeger.dev.storj.io/trace/3c902a745058257d FAIL (2025-12-08 21:32:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.507950983s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 1.986720538s trace: https://jaeger.dev.storj.io/trace/7773f135abca008 FAIL (2025-12-08 21:32:26 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.986749643s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.688754535s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:32:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.688784874s core uplink us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 468.984538ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/51c3a25722f21601 FAIL (2025-12-08 21:32:10 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 469.012702ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.672855071s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:31:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.672890918s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [........X......................X....X.X..................X..] duration: 3.500526291s trace: https://jaeger.dev.storj.io/trace/3c902a745058257d FAIL (2025-12-08 21:31:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.500546869s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.020606915s trace: https://jaeger.dev.storj.io/trace/7773f135abca008 FAIL (2025-12-08 21:31:40 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.020639798s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..................X.................X..X......X.....X..X....] duration: 5.882716354s trace: https://jaeger.dev.storj.io/trace/627c87facdf73edb FAIL (2025-12-08 21:31:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.882738916s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.698057847s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:31:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.698086232s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.......X.........X..................] duration: 5.732846629s trace: https://jaeger.dev.storj.io/trace/5fd4c7e9abd66483 FAIL (2025-12-08 21:31:23 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.732873471s core uplink us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 478.6743ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/51c3a25722f21601 FAIL (2025-12-08 21:31:11 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 478.704998ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.........X......................X....X.X..................X.] duration: 3.435108686s trace: https://jaeger.dev.storj.io/trace/3c902a745058257d FAIL (2025-12-08 21:31:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.435131079s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X.......X.........X.................] duration: 5.874441202s trace: https://jaeger.dev.storj.io/trace/5fd4c7e9abd66483 FAIL (2025-12-08 21:30:58 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.874494734s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.............................X.............................] duration: 5.645721727s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:30:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.645749138s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.004489824s trace: https://jaeger.dev.storj.io/trace/7773f135abca008 FAIL (2025-12-08 21:30:53 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.004518246s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...................X.................X..X......X.....X..X...] duration: 5.79361181s trace: https://jaeger.dev.storj.io/trace/627c87facdf73edb FAIL (2025-12-08 21:30:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.793640754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.............................X............................] duration: 5.713967363s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:30:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.71399804s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.024293244s trace: https://jaeger.dev.storj.io/trace/7773f135abca008 FAIL (2025-12-08 21:30:09 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.024317971s core uplink us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 473.610215ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/51c3a25722f21601 FAIL (2025-12-08 21:29:56 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 473.640323ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..........X......................X....X.X..................X] duration: 5.001010313s trace: https://jaeger.dev.storj.io/trace/3c902a745058257d FAIL (2025-12-08 21:29:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001033398s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [....................X.................X..X......X.....X..X..] duration: 5.828744468s trace: https://jaeger.dev.storj.io/trace/627c87facdf73edb FAIL (2025-12-08 21:29:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.828772741s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 15.411005322s remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/51c3a25722f21601 FAIL (2025-12-08 21:29:53 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 15.411021482s core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.......X.........X................] duration: 5.677208157s trace: https://jaeger.dev.storj.io/trace/5fd4c7e9abd66483 FAIL (2025-12-08 21:29:52 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.677245817s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.............................X...........................] duration: 5.926985969s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:29:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.927008022s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.....................X.................X..X......X.....X..X.] duration: 5.949001042s trace: https://jaeger.dev.storj.io/trace/627c87facdf73edb FAIL (2025-12-08 21:29:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.94901602s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...........X......................X....X.X..................] duration: 3.418226071s trace: https://jaeger.dev.storj.io/trace/2b04536d9d1e74bd FAIL (2025-12-08 21:29:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.418245279s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [......................X.................X..X......X.....X..X] duration: 1m0.001809832s trace: https://jaeger.dev.storj.io/trace/627c87facdf73edb FAIL (2025-12-08 21:29:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001831895s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.............................X..........................] duration: 5.838410755s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:29:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.838461873s core uplink us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 467.608235ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/51c3a25722f21601 FAIL (2025-12-08 21:29:01 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 467.640357ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X.......X.........X...............] duration: 5.807860261s trace: https://jaeger.dev.storj.io/trace/5fd4c7e9abd66483 FAIL (2025-12-08 21:29:00 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.807889807s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.............................X.........................] duration: 5.735316646s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:28:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.735346412s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.031374327s trace: https://jaeger.dev.storj.io/trace/7773f135abca008 FAIL (2025-12-08 21:28:54 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.031406799s core uplink us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 475.961913ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/51c3a25722f21601 FAIL (2025-12-08 21:28:44 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 475.984836ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [............X......................X....X.X.................] duration: 3.435499652s trace: https://jaeger.dev.storj.io/trace/2b04536d9d1e74bd FAIL (2025-12-08 21:28:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.435530049s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.......................X.................X..X......X.....X..] duration: 5.871312677s trace: https://jaeger.dev.storj.io/trace/b15c059e57dc6d8 FAIL (2025-12-08 21:28:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.87135091s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.000044018s trace: https://jaeger.dev.storj.io/trace/7773f135abca008 FAIL (2025-12-08 21:28:07 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.000074706s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X.......X.........X..............] duration: 5.85773301s trace: https://jaeger.dev.storj.io/trace/5fd4c7e9abd66483 FAIL (2025-12-08 21:28:02 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.857761895s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.............................X........................] duration: 5.699665731s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:27:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.699693752s core uplink us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.484743098s remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/51c3a25722f21601 FAIL (2025-12-08 21:27:46 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 1.484777604s core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X.......X.........X.............] duration: 5.710892208s trace: https://jaeger.dev.storj.io/trace/5fd4c7e9abd66483 FAIL (2025-12-08 21:27:36 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.710923446s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.............X......................X....X.X................] duration: 3.483272593s trace: https://jaeger.dev.storj.io/trace/2b04536d9d1e74bd FAIL (2025-12-08 21:27:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.483340111s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.............................X.......................] duration: 5.77683324s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:27:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.776858748s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [........................X.................X..X......X.....X.] duration: 5.884156514s trace: https://jaeger.dev.storj.io/trace/b15c059e57dc6d8 FAIL (2025-12-08 21:27:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.884173357s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.009326375s trace: https://jaeger.dev.storj.io/trace/7773f135abca008 FAIL (2025-12-08 21:27:08 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.009354477s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.........................X.................X..X......X.....X] duration: 1m0.000469478s trace: https://jaeger.dev.storj.io/trace/b15c059e57dc6d8 FAIL (2025-12-08 21:27:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000493953s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 467.279099ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/51c3a25722f21601 FAIL (2025-12-08 21:27:01 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 467.309508ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X.......X.........X............] duration: 5.781468929s trace: https://jaeger.dev.storj.io/trace/5fd4c7e9abd66483 FAIL (2025-12-08 21:27:00 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.781497244s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..............X......................X....X.X...............] duration: 3.506433774s trace: https://jaeger.dev.storj.io/trace/2b04536d9d1e74bd FAIL (2025-12-08 21:26:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.506461186s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X.......X.........X...........] duration: 6.180224055s trace: https://jaeger.dev.storj.io/trace/5fd4c7e9abd66483 FAIL (2025-12-08 21:26:41 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.180253612s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.............................X......................] duration: 5.733756877s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:26:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.733791944s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)