Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...........................X......................X....X.X..] duration: 3.461390748s trace: https://jaeger.dev.storj.io/trace/2b04536d9d1e74bd FAIL (2025-12-08 21:16:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.461458427s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.009580385s trace: https://jaeger.dev.storj.io/trace/7773f135abca008 FAIL (2025-12-08 21:16:29 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.009627092s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.............................X.........] duration: 5.76875078s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:16:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.768799783s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X.................X..] duration: 5.861172815s trace: https://jaeger.dev.storj.io/trace/4578ce8acdd2d76c FAIL (2025-12-08 21:16:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.861264299s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.989032805s trace: https://jaeger.dev.storj.io/trace/7773f135abca008 FAIL (2025-12-08 21:16:13 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.9890611s core uplink us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 468.257249ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/51c3a25722f21601 FAIL (2025-12-08 21:15:52 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 468.287255ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [............................X......................X....X.X.] duration: 3.532348273s trace: https://jaeger.dev.storj.io/trace/2b04536d9d1e74bd FAIL (2025-12-08 21:15:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.532378519s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.............................X........] duration: 5.827301309s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:15:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.827331326s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................................X.................X.] duration: 5.762117645s trace: https://jaeger.dev.storj.io/trace/4578ce8acdd2d76c FAIL (2025-12-08 21:15:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.762134868s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.................X] duration: 1m0.000705015s trace: https://jaeger.dev.storj.io/trace/4578ce8acdd2d76c FAIL (2025-12-08 21:15:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000794226s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.............................X.......] duration: 5.768082614s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:15:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.768119784s core uplink us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 467.93075ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/51c3a25722f21601 FAIL (2025-12-08 21:15:10 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 467.958221ms core satellite eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 30.887824823s remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/51c3a25722f21601 FAIL (2025-12-08 21:15:09 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 30.887854721s core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 1.978077125s trace: https://jaeger.dev.storj.io/trace/7773f135abca008 FAIL (2025-12-08 21:15:07 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.978099076s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.............................X......................X....X.X] duration: 5.001288308s trace: https://jaeger.dev.storj.io/trace/2b04536d9d1e74bd FAIL (2025-12-08 21:14:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001315138s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.............................X......] duration: 5.704190249s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:14:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.704225958s core uplink us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 476.198613ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/51c3a25722f21601 FAIL (2025-12-08 21:14:06 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 476.234731ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............................X......................X....X.] duration: 3.50535426s trace: https://jaeger.dev.storj.io/trace/343b2d2c590c10e0 FAIL (2025-12-08 21:14:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.505382524s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.001272865s trace: https://jaeger.dev.storj.io/trace/7773f135abca008 FAIL (2025-12-08 21:13:55 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.001303383s core uplink us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 469.446396ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/51c3a25722f21601 FAIL (2025-12-08 21:13:36 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 469.463699ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............................X......................X....X] duration: 5.001085846s trace: https://jaeger.dev.storj.io/trace/343b2d2c590c10e0 FAIL (2025-12-08 21:13:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001109962s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.805606288s trace: https://jaeger.dev.storj.io/trace/2f1573fb99882f3d FAIL (2025-12-08 21:13:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.805663698s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................X......................X....] duration: 3.469286754s trace: https://jaeger.dev.storj.io/trace/7f4d8684b142e218 FAIL (2025-12-08 21:13:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.469314386s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.............................X.....] duration: 5.671458413s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:13:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.671507807s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.997302643s trace: https://jaeger.dev.storj.io/trace/7773f135abca008 FAIL (2025-12-08 21:12:51 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.997330987s core uplink us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 468.113157ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/51c3a25722f21601 FAIL (2025-12-08 21:12:38 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 468.138554ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.888424388s trace: https://jaeger.dev.storj.io/trace/2f1573fb99882f3d FAIL (2025-12-08 21:12:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.88850465s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.............................X....] duration: 5.624558073s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:12:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.624684214s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................X......................X...] duration: 3.49112387s trace: https://jaeger.dev.storj.io/trace/7f4d8684b142e218 FAIL (2025-12-08 21:12:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.491190067s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.998965278s trace: https://jaeger.dev.storj.io/trace/7773f135abca008 FAIL (2025-12-08 21:12:05 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.998999082s core uplink us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 467.028972ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/51c3a25722f21601 FAIL (2025-12-08 21:12:03 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 467.055573ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.902598366s trace: https://jaeger.dev.storj.io/trace/2f1573fb99882f3d FAIL (2025-12-08 21:11:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.902629054s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.............................X...] duration: 5.727456212s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:11:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.727490667s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................X......................X..] duration: 3.40638297s trace: https://jaeger.dev.storj.io/trace/7f4d8684b142e218 FAIL (2025-12-08 21:11:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.40640946s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 1.957268269s trace: https://jaeger.dev.storj.io/trace/7773f135abca008 FAIL (2025-12-08 21:11:14 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.957297345s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.667479811s trace: https://jaeger.dev.storj.io/trace/2f1573fb99882f3d FAIL (2025-12-08 21:11:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.667548391s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 1.986697641s trace: https://jaeger.dev.storj.io/trace/7773f135abca008 FAIL (2025-12-08 21:10:58 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.98671825s core uplink us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 466.920376ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/51c3a25722f21601 FAIL (2025-12-08 21:10:58 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 466.949791ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.............................X..] duration: 5.716597833s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:10:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.716625125s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 1.993128774s trace: https://jaeger.dev.storj.io/trace/7773f135abca008 FAIL (2025-12-08 21:10:35 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.993159102s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................................X......................X.] duration: 3.480220448s trace: https://jaeger.dev.storj.io/trace/7f4d8684b142e218 FAIL (2025-12-08 21:10:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.480243933s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.001831798s trace: https://jaeger.dev.storj.io/trace/7773f135abca008 FAIL (2025-12-08 21:10:18 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.001862738s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.857670815s trace: https://jaeger.dev.storj.io/trace/2f1573fb99882f3d FAIL (2025-12-08 21:10:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.857722684s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 479.842571ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/51c3a25722f21601 FAIL (2025-12-08 21:10:06 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 479.866247ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 1.9865411s trace: https://jaeger.dev.storj.io/trace/7773f135abca008 FAIL (2025-12-08 21:09:54 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.986570457s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................................X......................X] duration: 5.001363272s trace: https://jaeger.dev.storj.io/trace/7f4d8684b142e218 FAIL (2025-12-08 21:09:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001384191s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X.............................X.] duration: 5.641761895s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:09:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.641792583s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.............................X] duration: 4m24.937381711s trace: https://jaeger.dev.storj.io/trace/2db34ec3fe434bce FAIL (2025-12-08 21:09:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 21:04:53.129554277 (77h35m53.000637322s) [upload=67811]: Starting upload Dec 8 21:04:53.129615013 (77h35m53.000698058s) [upload=67811]: Done starting upload Dec 8 21:04:53.129845151 (77h35m53.000928196s) [upload=67811]: Uploading segments... Dec 8 21:04:53.129948197 (77h35m53.001031243s) [upload=67811]: Got next segment. Inline: false Dec 8 21:04:53.129991991 (77h35m53.001075036s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 21:04:53.129998843 (77h35m53.001081879s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 21:04:54.262279080 (77h35m54.133362135s) [seg_pos={0 0}, upload=67811]: Begin upload segment... Dec 8 21:04:54.262341829 (77h35m54.133424884s) [seg_pos={0 0}, upload=67811]: Done begin upload segment. Dec 8 21:04:54.262356968 (77h35m54.133440003s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262437392 (77h35m54.133520437s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.262897227 (77h35m54.133980262s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263306957 (77h35m54.134390003s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.263701088 (77h35m54.134784133s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264096531 (77h35m54.135179566s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264512343 (77h35m54.135595388s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.264911823 (77h35m54.135994868s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265365146 (77h35m54.136448182s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.265906778 (77h35m54.136989823s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.267452732 (77h35m54.138535767s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.269003896 (77h35m54.140086941s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.270557483 (77h35m54.141640518s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.272052961 (77h35m54.143136016s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.273534212 (77h35m54.144617257s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:54.275143206 (77h35m54.146226241s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Uploading piece... Dec 8 21:04:55.967221223 (77h35m55.838304278s) [upload=67811]: Next returned nil segment Dec 8 21:04:55.967237525 (77h35m55.83832057s) [upload=67811]: Waiting for error group managing segments... Dec 8 21:04:56.251969629 (77h35m56.123052674s) [noise=true , address=145.79.227.24:4581 , piece_id=SOT7V3OGCK5F7ZHXE4XL5JEXXJCPJFL4AKKUTJWKAKJM4GI3TNMQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.255527102 (77h35m56.126610137s) [noise=true , address=145.79.227.32:4834 , piece_id=LN75KMAVDCY6UKRSVHBHSLMMLPFT7TTXWCTTELJ3QXM43JKKPCRQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.256885629 (77h35m56.127968664s) [noise=true , address=145.79.227.26:4654 , piece_id=7IU5GSOQAUBT3BAAFRC2ZB7BNWYVUKGDIBKMBL4Z3QWJBIABWXVA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.259886082 (77h35m56.130969117s) [noise=true , address=145.79.227.39:4015 , piece_id=YMGXVFFMVMHTGRPO2CHS2EDVJW3HE7GQF4F5PVMXZJ3FDU4IHT4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.260387237 (77h35m56.131470272s) [noise=true , address=145.79.227.26:4653 , piece_id=QYJZSNIRAFBTXYJYW3SAAI6AVWLPBSRHIA5NQAY5PUQEZMS3LMEA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.263362913 (77h35m56.134445958s) [noise=true , address=145.79.227.11:4732 , piece_id=VGNGWM5JQZL66YEINF57CYTTWMLMRAGXG3RDBSIAL5MHU2CV3Q4A, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.279894463 (77h35m56.150977508s) [noise=true , address=145.79.227.21:4475 , piece_id=CFBVY77PI5V6CBGW3OANJ5RYB6WFVIB6LOKRFGVLIPSDCQG2KKSQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282211485 (77h35m56.15329453s) [noise=true , address=145.79.227.43:4157 , piece_id=DH5CKGTBVFFKPBG5M4M6T2AZS2QJF4URT6NVNKETLSBBYFLYFZGQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282784135 (77h35m56.153867181s) [noise=true , address=145.79.227.20:4438 , piece_id=AGY7FMQM7PXLOZA5WDE4LW4YSJKHR5TPM6SIMAJDNWNT2SLMWWPQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.282875890 (77h35m56.153958925s) [noise=true , address=145.79.227.41:4085 , piece_id=PVAXRFP5YYNVS3Q5XRUCDKKBMUVLQD5UD3ZSF6ZSGX7VCLVRDSFQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.285674970 (77h35m56.156758005s) [noise=true , address=145.79.227.38:4978 , piece_id=Y3BJGNQPDFSS2RQZETOFUMOMTNDSOJ4Z2MQOOMHBXLPS5JGHP5XQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.288440746 (77h35m56.159523791s) [noise=true , address=145.79.227.20:4437 , piece_id=I2BVDHEQZ346BU32OY6ICEDONHID4UKTZJRCFVU6LJHTJAACATFA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289408038 (77h35m56.160491083s) [noise=true , address=145.79.227.34:4906 , piece_id=YHX5KO5HUHAYAAQDXROHGALPYLFNAQTDUFMM6SMHUAFC6XYX4HUA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: <nil> Dec 8 21:04:56.289431483 (77h35m56.160514518s) [seg_pos={0 0}, upload=67811]: Segment reached optimal threshold of 13 pieces. Dec 8 21:04:56.289656010 (77h35m56.160739045s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53718->77.90.31.18:4806: use of closed network connection Dec 8 21:04:56.289679105 (77h35m56.16076214s) [noise=true , address=77.90.31.18:4806 , piece_id=JXMY2LCYPGVXXRVQE4FHA52SO7FYKOTGYYVAGMJXZQJRUDH2Q6EA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289848216 (77h35m56.160931261s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289858967 (77h35m56.160942002s) [noise=true , address=77.90.31.11:4553 , piece_id=5HIQF4XMRQAWP2RMX7IUHO76RKLLFYVJRFDB2X6EKJGDRQ6HJDBA, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289944239 (77h35m56.161027284s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 8 21:04:56.289952244 (77h35m56.161035299s) [noise=true , address=77.90.31.21:4877 , piece_id=5DJWXRE27GLEOSO2VDKS4VI2R4B2TYMDBLS6CVCGQHF6VHENHZTQ, seg_pos={0 0}, upload=67811]: Long tail cancellation due to optimal threshold Dec 8 21:04:56.289970259 (77h35m56.161053314s) [seg_pos={0 0}, upload=67811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 21:04:56.290061623 (77h35m56.161144688s) [upload=67811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 21:04:56.290087081 (77h35m56.161170156s) [upload=67811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 21:09:16.202460324 (77h40m16.073543369s) [upload=67811]: Done uploading segments... Dec 8 21:09:16.202528694 (77h40m16.073611739s) [upload=67811]: Upload finished. err: metaclient: context deadline exceeded Duration: 4m24.937406598s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.027850143s trace: https://jaeger.dev.storj.io/trace/2f1573fb99882f3d FAIL (2025-12-08 21:09:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.027872334s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.......................................X.......X.........] duration: 5.996470467s trace: https://jaeger.dev.storj.io/trace/77d09bc4a21132bb FAIL (2025-12-08 21:09:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.996500264s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)