Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X..............................XX...] duration: 7.382079362s trace: https://jaeger.dev.storj.io/trace/21e18bb10e952a93 FAIL (2025-12-08 08:27:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.382104519s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................................X..X................] duration: 2.125389081s trace: https://jaeger.dev.storj.io/trace/23e4fea4d1c3bcb8 FAIL (2025-12-08 08:27:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 08:16:58.813434894 (64h47m58.684517929s) [upload=56813]: Starting upload Dec 8 08:16:58.813552538 (64h47m58.684635583s) [upload=56813]: Done starting upload Dec 8 08:16:58.813656476 (64h47m58.684739521s) [upload=56813]: Uploading segments... Dec 8 08:16:58.813718364 (64h47m58.684801409s) [upload=56813]: Got next segment. Inline: true Dec 8 08:16:58.813724626 (64h47m58.684807661s) [upload=56813]: Waiting for error group managing segments... Dec 8 08:16:58.813831950 (64h47m58.684914995s) [upload=56813]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 08:16:58.813839474 (64h47m58.68492254s) [upload=56813]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 08:16:58.813843732 (64h47m58.684926777s) [upload=56813]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 08:17:25.151880477 (64h48m25.022963512s) [upload=56813]: Done uploading segments... Dec 8 08:17:25.151931593 (64h48m25.023014638s) [upload=56813]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.125627897s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......................X........................X......X....] duration: 3.745929515s trace: https://jaeger.dev.storj.io/trace/5713ceca9b4db6c3 FAIL (2025-12-08 08:27:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.74595846s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 8.49430707s trace: https://jaeger.dev.storj.io/trace/6fbc0248558ad673 FAIL (2025-12-08 08:27:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 07:51:49.892373053 (64h22m49.763456099s) [upload=56436]: Starting upload Dec 8 07:51:49.892470558 (64h22m49.763553603s) [upload=56436]: Done starting upload Dec 8 07:51:49.892577121 (64h22m49.763660176s) [upload=56436]: Uploading segments... Dec 8 07:51:49.892739571 (64h22m49.763822606s) [upload=56436]: Got next segment. Inline: false Dec 8 07:51:49.892803042 (64h22m49.763886087s) [upload=56436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 07:51:49.892810626 (64h22m49.763893671s) [upload=56436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 07:51:50.139750947 (64h22m50.010833992s) [seg_pos={0 0}, upload=56436]: Begin upload segment... Dec 8 07:51:50.139806774 (64h22m50.010889819s) [seg_pos={0 0}, upload=56436]: Done begin upload segment. Dec 8 07:51:50.139840288 (64h22m50.010923333s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.139892166 (64h22m50.010975211s) [noise=true , address=145.79.227.24:4582 , piece_id=ZVY2X2XMRLDJA74AL4UMYY5A6MCEO4L2GUI4G77CXHM54CL6XTCA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.140602369 (64h22m50.011685414s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.141877286 (64h22m50.012960331s) [noise=true , address=145.79.227.23:4547 , piece_id=BCYPKLT75XINURN3XAXXBALSQOHSYB4PIUEVAOLI4OGBUSEPGKOA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.142886528 (64h22m50.013969563s) [noise=true , address=145.79.227.26:4655 , piece_id=KAK4VED4HFJXM6WO3ICYAUBWVHCQRKW4ACETAGAZYCHKJCLV6JRQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.143398483 (64h22m50.014481528s) [noise=true , address=145.79.227.42:4122 , piece_id=5YUJVB6L5LYLRBJ2475TVR5UY7NWIWTOE7KLKQJAXKOZRTGXL5IQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.143786251 (64h22m50.014869296s) [noise=true , address=145.79.227.22:4510 , piece_id=MURUKN5W3JWOHKZGIDJ3NAF5TCHFOVM2A46QFIMLUP6JKQXGZIWQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.144207654 (64h22m50.015290699s) [noise=true , address=77.90.31.10:4519 , piece_id=BWYJYFKLJPOXDUYYNGS6OXZ4N3J62CRLSRKMIDBJNEUZB54IWEKQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.144614459 (64h22m50.015697494s) [noise=true , address=145.79.227.20:4439 , piece_id=3AQ7WAXEFXGY6VGOQQACWGUGWDUDRAYIWXUQFBSZXWHFPVLMBIWQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.145229349 (64h22m50.016312394s) [noise=true , address=145.79.227.44:4194 , piece_id=7HLBO2SUUG5ZFYGMTMWZ4XMN5VO4LQRVVTNQ46CZOBU3MB32EO6A, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.146391893 (64h22m50.017474938s) [noise=true , address=145.79.227.39:4015 , piece_id=4TBSZGVUKZRXFHSU3C2NSJ2H26ZKMMURPNJFEQGS2ICQI6KZUNKQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.146973060 (64h22m50.018056105s) [noise=true , address=77.90.31.14:4663 , piece_id=KRLEX4OSLJJVKHVEOVAQALC374PIN3SK2L6BM63OVCAWZCS5NIHQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.147665980 (64h22m50.018749025s) [noise=true , address=145.79.227.25:4618 , piece_id=5NBPSN653OJAOTT5SQYUCHASNWGR7EFCFIAMUHYH3M7VILCUIAHQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.148271602 (64h22m50.019354637s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.149087727 (64h22m50.020170772s) [noise=true , address=45.91.116.5:4297 , piece_id=JYAFY55C5EYMVOYCF3LY25UKALYL76NR5SN5DKH3RB67WKGR62WQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.151023432 (64h22m50.022106477s) [noise=true , address=145.79.227.28:4727 , piece_id=XHE4AX4UFAA53UOTMI5TDTA2GQXM5QMWQDVKSGUGOQIXF6JA7PVA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:51.761342963 (64h22m51.632426008s) [upload=56436]: Next returned nil segment Dec 8 07:51:51.761356038 (64h22m51.632439083s) [upload=56436]: Waiting for error group managing segments... Dec 8 07:51:52.143112877 (64h22m52.014195912s) [noise=true , address=145.79.227.42:4122 , piece_id=5YUJVB6L5LYLRBJ2475TVR5UY7NWIWTOE7KLKQJAXKOZRTGXL5IQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.143778344 (64h22m52.014861389s) [noise=true , address=145.79.227.25:4618 , piece_id=5NBPSN653OJAOTT5SQYUCHASNWGR7EFCFIAMUHYH3M7VILCUIAHQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.146079326 (64h22m52.017162371s) [noise=true , address=145.79.227.20:4439 , piece_id=3AQ7WAXEFXGY6VGOQQACWGUGWDUDRAYIWXUQFBSZXWHFPVLMBIWQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.146479507 (64h22m52.017562542s) [noise=true , address=145.79.227.22:4510 , piece_id=MURUKN5W3JWOHKZGIDJ3NAF5TCHFOVM2A46QFIMLUP6JKQXGZIWQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.155737106 (64h22m52.026820151s) [noise=true , address=145.79.227.39:4015 , piece_id=4TBSZGVUKZRXFHSU3C2NSJ2H26ZKMMURPNJFEQGS2ICQI6KZUNKQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.161614518 (64h22m52.032697553s) [noise=true , address=145.79.227.23:4547 , piece_id=BCYPKLT75XINURN3XAXXBALSQOHSYB4PIUEVAOLI4OGBUSEPGKOA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.166235817 (64h22m52.037318862s) [noise=true , address=145.79.227.28:4727 , piece_id=XHE4AX4UFAA53UOTMI5TDTA2GQXM5QMWQDVKSGUGOQIXF6JA7PVA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.274077770 (64h22m52.145160815s) [noise=true , address=77.90.31.14:4663 , piece_id=KRLEX4OSLJJVKHVEOVAQALC374PIN3SK2L6BM63OVCAWZCS5NIHQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.311177544 (64h22m52.182260579s) [noise=true , address=45.91.116.5:4297 , piece_id=JYAFY55C5EYMVOYCF3LY25UKALYL76NR5SN5DKH3RB67WKGR62WQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.352570024 (64h22m52.223653069s) [noise=true , address=77.90.31.10:4519 , piece_id=BWYJYFKLJPOXDUYYNGS6OXZ4N3J62CRLSRKMIDBJNEUZB54IWEKQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.493961551 (64h22m52.365044586s) [noise=true , address=145.79.227.24:4582 , piece_id=ZVY2X2XMRLDJA74AL4UMYY5A6MCEO4L2GUI4G77CXHM54CL6XTCA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.494609204 (64h22m52.365692249s) [noise=true , address=145.79.227.44:4194 , piece_id=7HLBO2SUUG5ZFYGMTMWZ4XMN5VO4LQRVVTNQ46CZOBU3MB32EO6A, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.522301065 (64h22m52.3933841s) [noise=true , address=145.79.227.26:4655 , piece_id=KAK4VED4HFJXM6WO3ICYAUBWVHCQRKW4ACETAGAZYCHKJCLV6JRQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.522323438 (64h22m52.393406483s) [seg_pos={0 0}, upload=56436]: Segment reached optimal threshold of 13 pieces. Dec 8 07:51:52.522550460 (64h22m52.393633495s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 8 07:51:52.522558455 (64h22m52.3936415s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522657554 (64h22m52.393740599s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection Dec 8 07:51:52.522668415 (64h22m52.39375145s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522900286 (64h22m52.393983331s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: expected piece hash; context canceled; EOF Dec 8 07:51:52.522956193 (64h22m52.394039258s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522986951 (64h22m52.394069996s) [seg_pos={0 0}, upload=56436]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 07:51:52.523097702 (64h22m52.394180737s) [upload=56436]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 07:51:52.523136586 (64h22m52.394219631s) [upload=56436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 07:52:28.774033712 (64h23m28.645116757s) [upload=56436]: Done uploading segments... Dec 8 07:52:28.774096702 (64h23m28.645179747s) [upload=56436]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.494345214s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X..X...............] duration: 3.067484863s trace: https://jaeger.dev.storj.io/trace/23e4fea4d1c3bcb8 FAIL (2025-12-08 08:27:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 08:16:58.813434894 (64h47m58.684517929s) [upload=56813]: Starting upload Dec 8 08:16:58.813552538 (64h47m58.684635583s) [upload=56813]: Done starting upload Dec 8 08:16:58.813656476 (64h47m58.684739521s) [upload=56813]: Uploading segments... Dec 8 08:16:58.813718364 (64h47m58.684801409s) [upload=56813]: Got next segment. Inline: true Dec 8 08:16:58.813724626 (64h47m58.684807661s) [upload=56813]: Waiting for error group managing segments... Dec 8 08:16:58.813831950 (64h47m58.684914995s) [upload=56813]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 08:16:58.813839474 (64h47m58.68492254s) [upload=56813]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 08:16:58.813843732 (64h47m58.684926777s) [upload=56813]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 08:17:25.151880477 (64h48m25.022963512s) [upload=56813]: Done uploading segments... Dec 8 08:17:25.151931593 (64h48m25.023014638s) [upload=56813]: Upload finished. err: metaclient: context deadline exceeded Duration:* 3.067512286s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 1.05051847s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/3eabec942a864841 FAIL (2025-12-08 08:27:04 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.050549087s core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...............XX.........X...............X.................] duration: 5.837624999s trace: https://jaeger.dev.storj.io/trace/396bcc0ee1169960 FAIL (2025-12-08 08:27:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.83765208s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X..............................XX..] duration: 7.037399555s trace: https://jaeger.dev.storj.io/trace/21e18bb10e952a93 FAIL (2025-12-08 08:26:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.037424534s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 4.84743776s trace: https://jaeger.dev.storj.io/trace/5699b43ed41f47e3 FAIL (2025-12-08 08:26:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.847458629s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.760088193s trace: https://jaeger.dev.storj.io/trace/775323a46ba4bd95 FAIL (2025-12-08 08:26:54 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.760180769s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 7.883459798s trace: https://jaeger.dev.storj.io/trace/6fbc0248558ad673 FAIL (2025-12-08 08:26:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 07:51:49.892373053 (64h22m49.763456099s) [upload=56436]: Starting upload Dec 8 07:51:49.892470558 (64h22m49.763553603s) [upload=56436]: Done starting upload Dec 8 07:51:49.892577121 (64h22m49.763660176s) [upload=56436]: Uploading segments... Dec 8 07:51:49.892739571 (64h22m49.763822606s) [upload=56436]: Got next segment. Inline: false Dec 8 07:51:49.892803042 (64h22m49.763886087s) [upload=56436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 07:51:49.892810626 (64h22m49.763893671s) [upload=56436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 07:51:50.139750947 (64h22m50.010833992s) [seg_pos={0 0}, upload=56436]: Begin upload segment... Dec 8 07:51:50.139806774 (64h22m50.010889819s) [seg_pos={0 0}, upload=56436]: Done begin upload segment. Dec 8 07:51:50.139840288 (64h22m50.010923333s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.139892166 (64h22m50.010975211s) [noise=true , address=145.79.227.24:4582 , piece_id=ZVY2X2XMRLDJA74AL4UMYY5A6MCEO4L2GUI4G77CXHM54CL6XTCA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.140602369 (64h22m50.011685414s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.141877286 (64h22m50.012960331s) [noise=true , address=145.79.227.23:4547 , piece_id=BCYPKLT75XINURN3XAXXBALSQOHSYB4PIUEVAOLI4OGBUSEPGKOA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.142886528 (64h22m50.013969563s) [noise=true , address=145.79.227.26:4655 , piece_id=KAK4VED4HFJXM6WO3ICYAUBWVHCQRKW4ACETAGAZYCHKJCLV6JRQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.143398483 (64h22m50.014481528s) [noise=true , address=145.79.227.42:4122 , piece_id=5YUJVB6L5LYLRBJ2475TVR5UY7NWIWTOE7KLKQJAXKOZRTGXL5IQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.143786251 (64h22m50.014869296s) [noise=true , address=145.79.227.22:4510 , piece_id=MURUKN5W3JWOHKZGIDJ3NAF5TCHFOVM2A46QFIMLUP6JKQXGZIWQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.144207654 (64h22m50.015290699s) [noise=true , address=77.90.31.10:4519 , piece_id=BWYJYFKLJPOXDUYYNGS6OXZ4N3J62CRLSRKMIDBJNEUZB54IWEKQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.144614459 (64h22m50.015697494s) [noise=true , address=145.79.227.20:4439 , piece_id=3AQ7WAXEFXGY6VGOQQACWGUGWDUDRAYIWXUQFBSZXWHFPVLMBIWQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.145229349 (64h22m50.016312394s) [noise=true , address=145.79.227.44:4194 , piece_id=7HLBO2SUUG5ZFYGMTMWZ4XMN5VO4LQRVVTNQ46CZOBU3MB32EO6A, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.146391893 (64h22m50.017474938s) [noise=true , address=145.79.227.39:4015 , piece_id=4TBSZGVUKZRXFHSU3C2NSJ2H26ZKMMURPNJFEQGS2ICQI6KZUNKQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.146973060 (64h22m50.018056105s) [noise=true , address=77.90.31.14:4663 , piece_id=KRLEX4OSLJJVKHVEOVAQALC374PIN3SK2L6BM63OVCAWZCS5NIHQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.147665980 (64h22m50.018749025s) [noise=true , address=145.79.227.25:4618 , piece_id=5NBPSN653OJAOTT5SQYUCHASNWGR7EFCFIAMUHYH3M7VILCUIAHQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.148271602 (64h22m50.019354637s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.149087727 (64h22m50.020170772s) [noise=true , address=45.91.116.5:4297 , piece_id=JYAFY55C5EYMVOYCF3LY25UKALYL76NR5SN5DKH3RB67WKGR62WQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.151023432 (64h22m50.022106477s) [noise=true , address=145.79.227.28:4727 , piece_id=XHE4AX4UFAA53UOTMI5TDTA2GQXM5QMWQDVKSGUGOQIXF6JA7PVA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:51.761342963 (64h22m51.632426008s) [upload=56436]: Next returned nil segment Dec 8 07:51:51.761356038 (64h22m51.632439083s) [upload=56436]: Waiting for error group managing segments... Dec 8 07:51:52.143112877 (64h22m52.014195912s) [noise=true , address=145.79.227.42:4122 , piece_id=5YUJVB6L5LYLRBJ2475TVR5UY7NWIWTOE7KLKQJAXKOZRTGXL5IQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.143778344 (64h22m52.014861389s) [noise=true , address=145.79.227.25:4618 , piece_id=5NBPSN653OJAOTT5SQYUCHASNWGR7EFCFIAMUHYH3M7VILCUIAHQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.146079326 (64h22m52.017162371s) [noise=true , address=145.79.227.20:4439 , piece_id=3AQ7WAXEFXGY6VGOQQACWGUGWDUDRAYIWXUQFBSZXWHFPVLMBIWQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.146479507 (64h22m52.017562542s) [noise=true , address=145.79.227.22:4510 , piece_id=MURUKN5W3JWOHKZGIDJ3NAF5TCHFOVM2A46QFIMLUP6JKQXGZIWQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.155737106 (64h22m52.026820151s) [noise=true , address=145.79.227.39:4015 , piece_id=4TBSZGVUKZRXFHSU3C2NSJ2H26ZKMMURPNJFEQGS2ICQI6KZUNKQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.161614518 (64h22m52.032697553s) [noise=true , address=145.79.227.23:4547 , piece_id=BCYPKLT75XINURN3XAXXBALSQOHSYB4PIUEVAOLI4OGBUSEPGKOA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.166235817 (64h22m52.037318862s) [noise=true , address=145.79.227.28:4727 , piece_id=XHE4AX4UFAA53UOTMI5TDTA2GQXM5QMWQDVKSGUGOQIXF6JA7PVA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.274077770 (64h22m52.145160815s) [noise=true , address=77.90.31.14:4663 , piece_id=KRLEX4OSLJJVKHVEOVAQALC374PIN3SK2L6BM63OVCAWZCS5NIHQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.311177544 (64h22m52.182260579s) [noise=true , address=45.91.116.5:4297 , piece_id=JYAFY55C5EYMVOYCF3LY25UKALYL76NR5SN5DKH3RB67WKGR62WQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.352570024 (64h22m52.223653069s) [noise=true , address=77.90.31.10:4519 , piece_id=BWYJYFKLJPOXDUYYNGS6OXZ4N3J62CRLSRKMIDBJNEUZB54IWEKQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.493961551 (64h22m52.365044586s) [noise=true , address=145.79.227.24:4582 , piece_id=ZVY2X2XMRLDJA74AL4UMYY5A6MCEO4L2GUI4G77CXHM54CL6XTCA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.494609204 (64h22m52.365692249s) [noise=true , address=145.79.227.44:4194 , piece_id=7HLBO2SUUG5ZFYGMTMWZ4XMN5VO4LQRVVTNQ46CZOBU3MB32EO6A, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.522301065 (64h22m52.3933841s) [noise=true , address=145.79.227.26:4655 , piece_id=KAK4VED4HFJXM6WO3ICYAUBWVHCQRKW4ACETAGAZYCHKJCLV6JRQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.522323438 (64h22m52.393406483s) [seg_pos={0 0}, upload=56436]: Segment reached optimal threshold of 13 pieces. Dec 8 07:51:52.522550460 (64h22m52.393633495s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 8 07:51:52.522558455 (64h22m52.3936415s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522657554 (64h22m52.393740599s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection Dec 8 07:51:52.522668415 (64h22m52.39375145s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522900286 (64h22m52.393983331s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: expected piece hash; context canceled; EOF Dec 8 07:51:52.522956193 (64h22m52.394039258s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522986951 (64h22m52.394069996s) [seg_pos={0 0}, upload=56436]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 07:51:52.523097702 (64h22m52.394180737s) [upload=56436]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 07:51:52.523136586 (64h22m52.394219631s) [upload=56436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 07:52:28.774033712 (64h23m28.645116757s) [upload=56436]: Done uploading segments... Dec 8 07:52:28.774096702 (64h23m28.645179747s) [upload=56436]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.883489584s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X..............................XX.] duration: 5.978548905s trace: https://jaeger.dev.storj.io/trace/21e18bb10e952a93 FAIL (2025-12-08 08:26:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.978571096s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X...........X...............................................] duration: 8.504354862s trace: https://jaeger.dev.storj.io/trace/6fbc0248558ad673 FAIL (2025-12-08 08:26:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 07:51:49.892373053 (64h22m49.763456099s) [upload=56436]: Starting upload Dec 8 07:51:49.892470558 (64h22m49.763553603s) [upload=56436]: Done starting upload Dec 8 07:51:49.892577121 (64h22m49.763660176s) [upload=56436]: Uploading segments... Dec 8 07:51:49.892739571 (64h22m49.763822606s) [upload=56436]: Got next segment. Inline: false Dec 8 07:51:49.892803042 (64h22m49.763886087s) [upload=56436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 07:51:49.892810626 (64h22m49.763893671s) [upload=56436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 07:51:50.139750947 (64h22m50.010833992s) [seg_pos={0 0}, upload=56436]: Begin upload segment... Dec 8 07:51:50.139806774 (64h22m50.010889819s) [seg_pos={0 0}, upload=56436]: Done begin upload segment. Dec 8 07:51:50.139840288 (64h22m50.010923333s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.139892166 (64h22m50.010975211s) [noise=true , address=145.79.227.24:4582 , piece_id=ZVY2X2XMRLDJA74AL4UMYY5A6MCEO4L2GUI4G77CXHM54CL6XTCA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.140602369 (64h22m50.011685414s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.141877286 (64h22m50.012960331s) [noise=true , address=145.79.227.23:4547 , piece_id=BCYPKLT75XINURN3XAXXBALSQOHSYB4PIUEVAOLI4OGBUSEPGKOA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.142886528 (64h22m50.013969563s) [noise=true , address=145.79.227.26:4655 , piece_id=KAK4VED4HFJXM6WO3ICYAUBWVHCQRKW4ACETAGAZYCHKJCLV6JRQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.143398483 (64h22m50.014481528s) [noise=true , address=145.79.227.42:4122 , piece_id=5YUJVB6L5LYLRBJ2475TVR5UY7NWIWTOE7KLKQJAXKOZRTGXL5IQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.143786251 (64h22m50.014869296s) [noise=true , address=145.79.227.22:4510 , piece_id=MURUKN5W3JWOHKZGIDJ3NAF5TCHFOVM2A46QFIMLUP6JKQXGZIWQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.144207654 (64h22m50.015290699s) [noise=true , address=77.90.31.10:4519 , piece_id=BWYJYFKLJPOXDUYYNGS6OXZ4N3J62CRLSRKMIDBJNEUZB54IWEKQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.144614459 (64h22m50.015697494s) [noise=true , address=145.79.227.20:4439 , piece_id=3AQ7WAXEFXGY6VGOQQACWGUGWDUDRAYIWXUQFBSZXWHFPVLMBIWQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.145229349 (64h22m50.016312394s) [noise=true , address=145.79.227.44:4194 , piece_id=7HLBO2SUUG5ZFYGMTMWZ4XMN5VO4LQRVVTNQ46CZOBU3MB32EO6A, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.146391893 (64h22m50.017474938s) [noise=true , address=145.79.227.39:4015 , piece_id=4TBSZGVUKZRXFHSU3C2NSJ2H26ZKMMURPNJFEQGS2ICQI6KZUNKQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.146973060 (64h22m50.018056105s) [noise=true , address=77.90.31.14:4663 , piece_id=KRLEX4OSLJJVKHVEOVAQALC374PIN3SK2L6BM63OVCAWZCS5NIHQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.147665980 (64h22m50.018749025s) [noise=true , address=145.79.227.25:4618 , piece_id=5NBPSN653OJAOTT5SQYUCHASNWGR7EFCFIAMUHYH3M7VILCUIAHQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.148271602 (64h22m50.019354637s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.149087727 (64h22m50.020170772s) [noise=true , address=45.91.116.5:4297 , piece_id=JYAFY55C5EYMVOYCF3LY25UKALYL76NR5SN5DKH3RB67WKGR62WQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.151023432 (64h22m50.022106477s) [noise=true , address=145.79.227.28:4727 , piece_id=XHE4AX4UFAA53UOTMI5TDTA2GQXM5QMWQDVKSGUGOQIXF6JA7PVA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:51.761342963 (64h22m51.632426008s) [upload=56436]: Next returned nil segment Dec 8 07:51:51.761356038 (64h22m51.632439083s) [upload=56436]: Waiting for error group managing segments... Dec 8 07:51:52.143112877 (64h22m52.014195912s) [noise=true , address=145.79.227.42:4122 , piece_id=5YUJVB6L5LYLRBJ2475TVR5UY7NWIWTOE7KLKQJAXKOZRTGXL5IQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.143778344 (64h22m52.014861389s) [noise=true , address=145.79.227.25:4618 , piece_id=5NBPSN653OJAOTT5SQYUCHASNWGR7EFCFIAMUHYH3M7VILCUIAHQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.146079326 (64h22m52.017162371s) [noise=true , address=145.79.227.20:4439 , piece_id=3AQ7WAXEFXGY6VGOQQACWGUGWDUDRAYIWXUQFBSZXWHFPVLMBIWQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.146479507 (64h22m52.017562542s) [noise=true , address=145.79.227.22:4510 , piece_id=MURUKN5W3JWOHKZGIDJ3NAF5TCHFOVM2A46QFIMLUP6JKQXGZIWQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.155737106 (64h22m52.026820151s) [noise=true , address=145.79.227.39:4015 , piece_id=4TBSZGVUKZRXFHSU3C2NSJ2H26ZKMMURPNJFEQGS2ICQI6KZUNKQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.161614518 (64h22m52.032697553s) [noise=true , address=145.79.227.23:4547 , piece_id=BCYPKLT75XINURN3XAXXBALSQOHSYB4PIUEVAOLI4OGBUSEPGKOA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.166235817 (64h22m52.037318862s) [noise=true , address=145.79.227.28:4727 , piece_id=XHE4AX4UFAA53UOTMI5TDTA2GQXM5QMWQDVKSGUGOQIXF6JA7PVA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.274077770 (64h22m52.145160815s) [noise=true , address=77.90.31.14:4663 , piece_id=KRLEX4OSLJJVKHVEOVAQALC374PIN3SK2L6BM63OVCAWZCS5NIHQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.311177544 (64h22m52.182260579s) [noise=true , address=45.91.116.5:4297 , piece_id=JYAFY55C5EYMVOYCF3LY25UKALYL76NR5SN5DKH3RB67WKGR62WQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.352570024 (64h22m52.223653069s) [noise=true , address=77.90.31.10:4519 , piece_id=BWYJYFKLJPOXDUYYNGS6OXZ4N3J62CRLSRKMIDBJNEUZB54IWEKQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.493961551 (64h22m52.365044586s) [noise=true , address=145.79.227.24:4582 , piece_id=ZVY2X2XMRLDJA74AL4UMYY5A6MCEO4L2GUI4G77CXHM54CL6XTCA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.494609204 (64h22m52.365692249s) [noise=true , address=145.79.227.44:4194 , piece_id=7HLBO2SUUG5ZFYGMTMWZ4XMN5VO4LQRVVTNQ46CZOBU3MB32EO6A, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.522301065 (64h22m52.3933841s) [noise=true , address=145.79.227.26:4655 , piece_id=KAK4VED4HFJXM6WO3ICYAUBWVHCQRKW4ACETAGAZYCHKJCLV6JRQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.522323438 (64h22m52.393406483s) [seg_pos={0 0}, upload=56436]: Segment reached optimal threshold of 13 pieces. Dec 8 07:51:52.522550460 (64h22m52.393633495s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 8 07:51:52.522558455 (64h22m52.3936415s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522657554 (64h22m52.393740599s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection Dec 8 07:51:52.522668415 (64h22m52.39375145s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522900286 (64h22m52.393983331s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: expected piece hash; context canceled; EOF Dec 8 07:51:52.522956193 (64h22m52.394039258s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522986951 (64h22m52.394069996s) [seg_pos={0 0}, upload=56436]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 07:51:52.523097702 (64h22m52.394180737s) [upload=56436]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 07:51:52.523136586 (64h22m52.394219631s) [upload=56436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 07:52:28.774033712 (64h23m28.645116757s) [upload=56436]: Done uploading segments... Dec 8 07:52:28.774096702 (64h23m28.645179747s) [upload=56436]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.504384559s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.683148039s trace: https://jaeger.dev.storj.io/trace/775323a46ba4bd95 FAIL (2025-12-08 08:26:21 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.68316499s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X..X..............] duration: 2.200983252s trace: https://jaeger.dev.storj.io/trace/23e4fea4d1c3bcb8 FAIL (2025-12-08 08:26:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 08:16:58.813434894 (64h47m58.684517929s) [upload=56813]: Starting upload Dec 8 08:16:58.813552538 (64h47m58.684635583s) [upload=56813]: Done starting upload Dec 8 08:16:58.813656476 (64h47m58.684739521s) [upload=56813]: Uploading segments... Dec 8 08:16:58.813718364 (64h47m58.684801409s) [upload=56813]: Got next segment. Inline: true Dec 8 08:16:58.813724626 (64h47m58.684807661s) [upload=56813]: Waiting for error group managing segments... Dec 8 08:16:58.813831950 (64h47m58.684914995s) [upload=56813]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 08:16:58.813839474 (64h47m58.68492254s) [upload=56813]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 08:16:58.813843732 (64h47m58.684926777s) [upload=56813]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 08:17:25.151880477 (64h48m25.022963512s) [upload=56813]: Done uploading segments... Dec 8 08:17:25.151931593 (64h48m25.023014638s) [upload=56813]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.201008532s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 4.942205461s trace: https://jaeger.dev.storj.io/trace/5699b43ed41f47e3 FAIL (2025-12-08 08:26:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.942235668s core uplink eu1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 1.137183061s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/3eabec942a864841 FAIL (2025-12-08 08:26:18 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.137208439s core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........................X........................X......X...] duration: 3.408022339s trace: https://jaeger.dev.storj.io/trace/5713ceca9b4db6c3 FAIL (2025-12-08 08:26:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.408039904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 46.966614752s trace: https://jaeger.dev.storj.io/trace/775323a46ba4bd95 FAIL (2025-12-08 08:26:15 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 46.966674786s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X..............................XX] duration: 34.138024147s trace: https://jaeger.dev.storj.io/trace/21e18bb10e952a93 FAIL (2025-12-08 08:26:10 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 34.138039216s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X...........X..............................................] duration: 8.975555318s trace: https://jaeger.dev.storj.io/trace/6fbc0248558ad673 FAIL (2025-12-08 08:26:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 07:51:49.892373053 (64h22m49.763456099s) [upload=56436]: Starting upload Dec 8 07:51:49.892470558 (64h22m49.763553603s) [upload=56436]: Done starting upload Dec 8 07:51:49.892577121 (64h22m49.763660176s) [upload=56436]: Uploading segments... Dec 8 07:51:49.892739571 (64h22m49.763822606s) [upload=56436]: Got next segment. Inline: false Dec 8 07:51:49.892803042 (64h22m49.763886087s) [upload=56436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 07:51:49.892810626 (64h22m49.763893671s) [upload=56436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 07:51:50.139750947 (64h22m50.010833992s) [seg_pos={0 0}, upload=56436]: Begin upload segment... Dec 8 07:51:50.139806774 (64h22m50.010889819s) [seg_pos={0 0}, upload=56436]: Done begin upload segment. Dec 8 07:51:50.139840288 (64h22m50.010923333s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.139892166 (64h22m50.010975211s) [noise=true , address=145.79.227.24:4582 , piece_id=ZVY2X2XMRLDJA74AL4UMYY5A6MCEO4L2GUI4G77CXHM54CL6XTCA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.140602369 (64h22m50.011685414s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.141877286 (64h22m50.012960331s) [noise=true , address=145.79.227.23:4547 , piece_id=BCYPKLT75XINURN3XAXXBALSQOHSYB4PIUEVAOLI4OGBUSEPGKOA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.142886528 (64h22m50.013969563s) [noise=true , address=145.79.227.26:4655 , piece_id=KAK4VED4HFJXM6WO3ICYAUBWVHCQRKW4ACETAGAZYCHKJCLV6JRQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.143398483 (64h22m50.014481528s) [noise=true , address=145.79.227.42:4122 , piece_id=5YUJVB6L5LYLRBJ2475TVR5UY7NWIWTOE7KLKQJAXKOZRTGXL5IQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.143786251 (64h22m50.014869296s) [noise=true , address=145.79.227.22:4510 , piece_id=MURUKN5W3JWOHKZGIDJ3NAF5TCHFOVM2A46QFIMLUP6JKQXGZIWQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.144207654 (64h22m50.015290699s) [noise=true , address=77.90.31.10:4519 , piece_id=BWYJYFKLJPOXDUYYNGS6OXZ4N3J62CRLSRKMIDBJNEUZB54IWEKQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.144614459 (64h22m50.015697494s) [noise=true , address=145.79.227.20:4439 , piece_id=3AQ7WAXEFXGY6VGOQQACWGUGWDUDRAYIWXUQFBSZXWHFPVLMBIWQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.145229349 (64h22m50.016312394s) [noise=true , address=145.79.227.44:4194 , piece_id=7HLBO2SUUG5ZFYGMTMWZ4XMN5VO4LQRVVTNQ46CZOBU3MB32EO6A, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.146391893 (64h22m50.017474938s) [noise=true , address=145.79.227.39:4015 , piece_id=4TBSZGVUKZRXFHSU3C2NSJ2H26ZKMMURPNJFEQGS2ICQI6KZUNKQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.146973060 (64h22m50.018056105s) [noise=true , address=77.90.31.14:4663 , piece_id=KRLEX4OSLJJVKHVEOVAQALC374PIN3SK2L6BM63OVCAWZCS5NIHQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.147665980 (64h22m50.018749025s) [noise=true , address=145.79.227.25:4618 , piece_id=5NBPSN653OJAOTT5SQYUCHASNWGR7EFCFIAMUHYH3M7VILCUIAHQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.148271602 (64h22m50.019354637s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.149087727 (64h22m50.020170772s) [noise=true , address=45.91.116.5:4297 , piece_id=JYAFY55C5EYMVOYCF3LY25UKALYL76NR5SN5DKH3RB67WKGR62WQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.151023432 (64h22m50.022106477s) [noise=true , address=145.79.227.28:4727 , piece_id=XHE4AX4UFAA53UOTMI5TDTA2GQXM5QMWQDVKSGUGOQIXF6JA7PVA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:51.761342963 (64h22m51.632426008s) [upload=56436]: Next returned nil segment Dec 8 07:51:51.761356038 (64h22m51.632439083s) [upload=56436]: Waiting for error group managing segments... Dec 8 07:51:52.143112877 (64h22m52.014195912s) [noise=true , address=145.79.227.42:4122 , piece_id=5YUJVB6L5LYLRBJ2475TVR5UY7NWIWTOE7KLKQJAXKOZRTGXL5IQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.143778344 (64h22m52.014861389s) [noise=true , address=145.79.227.25:4618 , piece_id=5NBPSN653OJAOTT5SQYUCHASNWGR7EFCFIAMUHYH3M7VILCUIAHQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.146079326 (64h22m52.017162371s) [noise=true , address=145.79.227.20:4439 , piece_id=3AQ7WAXEFXGY6VGOQQACWGUGWDUDRAYIWXUQFBSZXWHFPVLMBIWQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.146479507 (64h22m52.017562542s) [noise=true , address=145.79.227.22:4510 , piece_id=MURUKN5W3JWOHKZGIDJ3NAF5TCHFOVM2A46QFIMLUP6JKQXGZIWQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.155737106 (64h22m52.026820151s) [noise=true , address=145.79.227.39:4015 , piece_id=4TBSZGVUKZRXFHSU3C2NSJ2H26ZKMMURPNJFEQGS2ICQI6KZUNKQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.161614518 (64h22m52.032697553s) [noise=true , address=145.79.227.23:4547 , piece_id=BCYPKLT75XINURN3XAXXBALSQOHSYB4PIUEVAOLI4OGBUSEPGKOA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.166235817 (64h22m52.037318862s) [noise=true , address=145.79.227.28:4727 , piece_id=XHE4AX4UFAA53UOTMI5TDTA2GQXM5QMWQDVKSGUGOQIXF6JA7PVA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.274077770 (64h22m52.145160815s) [noise=true , address=77.90.31.14:4663 , piece_id=KRLEX4OSLJJVKHVEOVAQALC374PIN3SK2L6BM63OVCAWZCS5NIHQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.311177544 (64h22m52.182260579s) [noise=true , address=45.91.116.5:4297 , piece_id=JYAFY55C5EYMVOYCF3LY25UKALYL76NR5SN5DKH3RB67WKGR62WQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.352570024 (64h22m52.223653069s) [noise=true , address=77.90.31.10:4519 , piece_id=BWYJYFKLJPOXDUYYNGS6OXZ4N3J62CRLSRKMIDBJNEUZB54IWEKQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.493961551 (64h22m52.365044586s) [noise=true , address=145.79.227.24:4582 , piece_id=ZVY2X2XMRLDJA74AL4UMYY5A6MCEO4L2GUI4G77CXHM54CL6XTCA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.494609204 (64h22m52.365692249s) [noise=true , address=145.79.227.44:4194 , piece_id=7HLBO2SUUG5ZFYGMTMWZ4XMN5VO4LQRVVTNQ46CZOBU3MB32EO6A, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.522301065 (64h22m52.3933841s) [noise=true , address=145.79.227.26:4655 , piece_id=KAK4VED4HFJXM6WO3ICYAUBWVHCQRKW4ACETAGAZYCHKJCLV6JRQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.522323438 (64h22m52.393406483s) [seg_pos={0 0}, upload=56436]: Segment reached optimal threshold of 13 pieces. Dec 8 07:51:52.522550460 (64h22m52.393633495s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 8 07:51:52.522558455 (64h22m52.3936415s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522657554 (64h22m52.393740599s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection Dec 8 07:51:52.522668415 (64h22m52.39375145s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522900286 (64h22m52.393983331s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: expected piece hash; context canceled; EOF Dec 8 07:51:52.522956193 (64h22m52.394039258s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522986951 (64h22m52.394069996s) [seg_pos={0 0}, upload=56436]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 07:51:52.523097702 (64h22m52.394180737s) [upload=56436]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 07:51:52.523136586 (64h22m52.394219631s) [upload=56436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 07:52:28.774033712 (64h23m28.645116757s) [upload=56436]: Done uploading segments... Dec 8 07:52:28.774096702 (64h23m28.645179747s) [upload=56436]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.975584705s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [................XX.........X...............X................] duration: 5.813929989s trace: https://jaeger.dev.storj.io/trace/396bcc0ee1169960 FAIL (2025-12-08 08:25:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.813987199s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.................XX.........X...............X...............] duration: 5.69476749s trace: https://jaeger.dev.storj.io/trace/396bcc0ee1169960 FAIL (2025-12-08 08:25:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.694782388s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................X........................X......X..] duration: 3.520569764s trace: https://jaeger.dev.storj.io/trace/5713ceca9b4db6c3 FAIL (2025-12-08 08:25:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.520647773s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X..............................X] duration: 9m50.453229277s trace: https://jaeger.dev.storj.io/trace/aeb976383aaf4c1 FAIL (2025-12-08 08:25:36 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 9m50.453250969s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X..X.............] duration: 2.133955713s trace: https://jaeger.dev.storj.io/trace/23e4fea4d1c3bcb8 FAIL (2025-12-08 08:25:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 08:16:58.813434894 (64h47m58.684517929s) [upload=56813]: Starting upload Dec 8 08:16:58.813552538 (64h47m58.684635583s) [upload=56813]: Done starting upload Dec 8 08:16:58.813656476 (64h47m58.684739521s) [upload=56813]: Uploading segments... Dec 8 08:16:58.813718364 (64h47m58.684801409s) [upload=56813]: Got next segment. Inline: true Dec 8 08:16:58.813724626 (64h47m58.684807661s) [upload=56813]: Waiting for error group managing segments... Dec 8 08:16:58.813831950 (64h47m58.684914995s) [upload=56813]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 08:16:58.813839474 (64h47m58.68492254s) [upload=56813]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 08:16:58.813843732 (64h47m58.684926777s) [upload=56813]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 08:17:25.151880477 (64h48m25.022963512s) [upload=56813]: Done uploading segments... Dec 8 08:17:25.151931593 (64h48m25.023014638s) [upload=56813]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.133981933s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.137803002s trace: https://jaeger.dev.storj.io/trace/5699b43ed41f47e3 FAIL (2025-12-08 08:25:14 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.137832799s core uplink eu1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 731.033812ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/3eabec942a864841 FAIL (2025-12-08 08:25:14 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 731.066404ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [............................................X..X............] duration: 2.225341613s trace: https://jaeger.dev.storj.io/trace/23e4fea4d1c3bcb8 FAIL (2025-12-08 08:25:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 08:16:58.813434894 (64h47m58.684517929s) [upload=56813]: Starting upload Dec 8 08:16:58.813552538 (64h47m58.684635583s) [upload=56813]: Done starting upload Dec 8 08:16:58.813656476 (64h47m58.684739521s) [upload=56813]: Uploading segments... Dec 8 08:16:58.813718364 (64h47m58.684801409s) [upload=56813]: Got next segment. Inline: true Dec 8 08:16:58.813724626 (64h47m58.684807661s) [upload=56813]: Waiting for error group managing segments... Dec 8 08:16:58.813831950 (64h47m58.684914995s) [upload=56813]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 08:16:58.813839474 (64h47m58.68492254s) [upload=56813]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 08:16:58.813843732 (64h47m58.684926777s) [upload=56813]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 08:17:25.151880477 (64h48m25.022963512s) [upload=56813]: Done uploading segments... Dec 8 08:17:25.151931593 (64h48m25.023014638s) [upload=56813]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.225373906s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X...........X.............................................] duration: 9.07257725s trace: https://jaeger.dev.storj.io/trace/6fbc0248558ad673 FAIL (2025-12-08 08:24:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 07:51:49.892373053 (64h22m49.763456099s) [upload=56436]: Starting upload Dec 8 07:51:49.892470558 (64h22m49.763553603s) [upload=56436]: Done starting upload Dec 8 07:51:49.892577121 (64h22m49.763660176s) [upload=56436]: Uploading segments... Dec 8 07:51:49.892739571 (64h22m49.763822606s) [upload=56436]: Got next segment. Inline: false Dec 8 07:51:49.892803042 (64h22m49.763886087s) [upload=56436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 07:51:49.892810626 (64h22m49.763893671s) [upload=56436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 07:51:50.139750947 (64h22m50.010833992s) [seg_pos={0 0}, upload=56436]: Begin upload segment... Dec 8 07:51:50.139806774 (64h22m50.010889819s) [seg_pos={0 0}, upload=56436]: Done begin upload segment. Dec 8 07:51:50.139840288 (64h22m50.010923333s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.139892166 (64h22m50.010975211s) [noise=true , address=145.79.227.24:4582 , piece_id=ZVY2X2XMRLDJA74AL4UMYY5A6MCEO4L2GUI4G77CXHM54CL6XTCA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.140602369 (64h22m50.011685414s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.141877286 (64h22m50.012960331s) [noise=true , address=145.79.227.23:4547 , piece_id=BCYPKLT75XINURN3XAXXBALSQOHSYB4PIUEVAOLI4OGBUSEPGKOA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.142886528 (64h22m50.013969563s) [noise=true , address=145.79.227.26:4655 , piece_id=KAK4VED4HFJXM6WO3ICYAUBWVHCQRKW4ACETAGAZYCHKJCLV6JRQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.143398483 (64h22m50.014481528s) [noise=true , address=145.79.227.42:4122 , piece_id=5YUJVB6L5LYLRBJ2475TVR5UY7NWIWTOE7KLKQJAXKOZRTGXL5IQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.143786251 (64h22m50.014869296s) [noise=true , address=145.79.227.22:4510 , piece_id=MURUKN5W3JWOHKZGIDJ3NAF5TCHFOVM2A46QFIMLUP6JKQXGZIWQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.144207654 (64h22m50.015290699s) [noise=true , address=77.90.31.10:4519 , piece_id=BWYJYFKLJPOXDUYYNGS6OXZ4N3J62CRLSRKMIDBJNEUZB54IWEKQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.144614459 (64h22m50.015697494s) [noise=true , address=145.79.227.20:4439 , piece_id=3AQ7WAXEFXGY6VGOQQACWGUGWDUDRAYIWXUQFBSZXWHFPVLMBIWQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.145229349 (64h22m50.016312394s) [noise=true , address=145.79.227.44:4194 , piece_id=7HLBO2SUUG5ZFYGMTMWZ4XMN5VO4LQRVVTNQ46CZOBU3MB32EO6A, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.146391893 (64h22m50.017474938s) [noise=true , address=145.79.227.39:4015 , piece_id=4TBSZGVUKZRXFHSU3C2NSJ2H26ZKMMURPNJFEQGS2ICQI6KZUNKQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.146973060 (64h22m50.018056105s) [noise=true , address=77.90.31.14:4663 , piece_id=KRLEX4OSLJJVKHVEOVAQALC374PIN3SK2L6BM63OVCAWZCS5NIHQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.147665980 (64h22m50.018749025s) [noise=true , address=145.79.227.25:4618 , piece_id=5NBPSN653OJAOTT5SQYUCHASNWGR7EFCFIAMUHYH3M7VILCUIAHQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.148271602 (64h22m50.019354637s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.149087727 (64h22m50.020170772s) [noise=true , address=45.91.116.5:4297 , piece_id=JYAFY55C5EYMVOYCF3LY25UKALYL76NR5SN5DKH3RB67WKGR62WQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.151023432 (64h22m50.022106477s) [noise=true , address=145.79.227.28:4727 , piece_id=XHE4AX4UFAA53UOTMI5TDTA2GQXM5QMWQDVKSGUGOQIXF6JA7PVA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:51.761342963 (64h22m51.632426008s) [upload=56436]: Next returned nil segment Dec 8 07:51:51.761356038 (64h22m51.632439083s) [upload=56436]: Waiting for error group managing segments... Dec 8 07:51:52.143112877 (64h22m52.014195912s) [noise=true , address=145.79.227.42:4122 , piece_id=5YUJVB6L5LYLRBJ2475TVR5UY7NWIWTOE7KLKQJAXKOZRTGXL5IQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.143778344 (64h22m52.014861389s) [noise=true , address=145.79.227.25:4618 , piece_id=5NBPSN653OJAOTT5SQYUCHASNWGR7EFCFIAMUHYH3M7VILCUIAHQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.146079326 (64h22m52.017162371s) [noise=true , address=145.79.227.20:4439 , piece_id=3AQ7WAXEFXGY6VGOQQACWGUGWDUDRAYIWXUQFBSZXWHFPVLMBIWQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.146479507 (64h22m52.017562542s) [noise=true , address=145.79.227.22:4510 , piece_id=MURUKN5W3JWOHKZGIDJ3NAF5TCHFOVM2A46QFIMLUP6JKQXGZIWQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.155737106 (64h22m52.026820151s) [noise=true , address=145.79.227.39:4015 , piece_id=4TBSZGVUKZRXFHSU3C2NSJ2H26ZKMMURPNJFEQGS2ICQI6KZUNKQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.161614518 (64h22m52.032697553s) [noise=true , address=145.79.227.23:4547 , piece_id=BCYPKLT75XINURN3XAXXBALSQOHSYB4PIUEVAOLI4OGBUSEPGKOA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.166235817 (64h22m52.037318862s) [noise=true , address=145.79.227.28:4727 , piece_id=XHE4AX4UFAA53UOTMI5TDTA2GQXM5QMWQDVKSGUGOQIXF6JA7PVA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.274077770 (64h22m52.145160815s) [noise=true , address=77.90.31.14:4663 , piece_id=KRLEX4OSLJJVKHVEOVAQALC374PIN3SK2L6BM63OVCAWZCS5NIHQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.311177544 (64h22m52.182260579s) [noise=true , address=45.91.116.5:4297 , piece_id=JYAFY55C5EYMVOYCF3LY25UKALYL76NR5SN5DKH3RB67WKGR62WQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.352570024 (64h22m52.223653069s) [noise=true , address=77.90.31.10:4519 , piece_id=BWYJYFKLJPOXDUYYNGS6OXZ4N3J62CRLSRKMIDBJNEUZB54IWEKQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.493961551 (64h22m52.365044586s) [noise=true , address=145.79.227.24:4582 , piece_id=ZVY2X2XMRLDJA74AL4UMYY5A6MCEO4L2GUI4G77CXHM54CL6XTCA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.494609204 (64h22m52.365692249s) [noise=true , address=145.79.227.44:4194 , piece_id=7HLBO2SUUG5ZFYGMTMWZ4XMN5VO4LQRVVTNQ46CZOBU3MB32EO6A, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.522301065 (64h22m52.3933841s) [noise=true , address=145.79.227.26:4655 , piece_id=KAK4VED4HFJXM6WO3ICYAUBWVHCQRKW4ACETAGAZYCHKJCLV6JRQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.522323438 (64h22m52.393406483s) [seg_pos={0 0}, upload=56436]: Segment reached optimal threshold of 13 pieces. Dec 8 07:51:52.522550460 (64h22m52.393633495s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 8 07:51:52.522558455 (64h22m52.3936415s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522657554 (64h22m52.393740599s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection Dec 8 07:51:52.522668415 (64h22m52.39375145s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522900286 (64h22m52.393983331s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: expected piece hash; context canceled; EOF Dec 8 07:51:52.522956193 (64h22m52.394039258s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522986951 (64h22m52.394069996s) [seg_pos={0 0}, upload=56436]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 07:51:52.523097702 (64h22m52.394180737s) [upload=56436]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 07:51:52.523136586 (64h22m52.394219631s) [upload=56436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 07:52:28.774033712 (64h23m28.645116757s) [upload=56436]: Done uploading segments... Dec 8 07:52:28.774096702 (64h23m28.645179747s) [upload=56436]: Upload finished. err: metaclient: context deadline exceeded Duration: 9.072609531s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 4.874273026s trace: https://jaeger.dev.storj.io/trace/5699b43ed41f47e3 FAIL (2025-12-08 08:24:46 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.87430121s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X...........X............................................] duration: 8.406520385s trace: https://jaeger.dev.storj.io/trace/6fbc0248558ad673 FAIL (2025-12-08 08:24:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 07:51:49.892373053 (64h22m49.763456099s) [upload=56436]: Starting upload Dec 8 07:51:49.892470558 (64h22m49.763553603s) [upload=56436]: Done starting upload Dec 8 07:51:49.892577121 (64h22m49.763660176s) [upload=56436]: Uploading segments... Dec 8 07:51:49.892739571 (64h22m49.763822606s) [upload=56436]: Got next segment. Inline: false Dec 8 07:51:49.892803042 (64h22m49.763886087s) [upload=56436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 07:51:49.892810626 (64h22m49.763893671s) [upload=56436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 07:51:50.139750947 (64h22m50.010833992s) [seg_pos={0 0}, upload=56436]: Begin upload segment... Dec 8 07:51:50.139806774 (64h22m50.010889819s) [seg_pos={0 0}, upload=56436]: Done begin upload segment. Dec 8 07:51:50.139840288 (64h22m50.010923333s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.139892166 (64h22m50.010975211s) [noise=true , address=145.79.227.24:4582 , piece_id=ZVY2X2XMRLDJA74AL4UMYY5A6MCEO4L2GUI4G77CXHM54CL6XTCA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.140602369 (64h22m50.011685414s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.141877286 (64h22m50.012960331s) [noise=true , address=145.79.227.23:4547 , piece_id=BCYPKLT75XINURN3XAXXBALSQOHSYB4PIUEVAOLI4OGBUSEPGKOA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.142886528 (64h22m50.013969563s) [noise=true , address=145.79.227.26:4655 , piece_id=KAK4VED4HFJXM6WO3ICYAUBWVHCQRKW4ACETAGAZYCHKJCLV6JRQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.143398483 (64h22m50.014481528s) [noise=true , address=145.79.227.42:4122 , piece_id=5YUJVB6L5LYLRBJ2475TVR5UY7NWIWTOE7KLKQJAXKOZRTGXL5IQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.143786251 (64h22m50.014869296s) [noise=true , address=145.79.227.22:4510 , piece_id=MURUKN5W3JWOHKZGIDJ3NAF5TCHFOVM2A46QFIMLUP6JKQXGZIWQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.144207654 (64h22m50.015290699s) [noise=true , address=77.90.31.10:4519 , piece_id=BWYJYFKLJPOXDUYYNGS6OXZ4N3J62CRLSRKMIDBJNEUZB54IWEKQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.144614459 (64h22m50.015697494s) [noise=true , address=145.79.227.20:4439 , piece_id=3AQ7WAXEFXGY6VGOQQACWGUGWDUDRAYIWXUQFBSZXWHFPVLMBIWQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.145229349 (64h22m50.016312394s) [noise=true , address=145.79.227.44:4194 , piece_id=7HLBO2SUUG5ZFYGMTMWZ4XMN5VO4LQRVVTNQ46CZOBU3MB32EO6A, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.146391893 (64h22m50.017474938s) [noise=true , address=145.79.227.39:4015 , piece_id=4TBSZGVUKZRXFHSU3C2NSJ2H26ZKMMURPNJFEQGS2ICQI6KZUNKQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.146973060 (64h22m50.018056105s) [noise=true , address=77.90.31.14:4663 , piece_id=KRLEX4OSLJJVKHVEOVAQALC374PIN3SK2L6BM63OVCAWZCS5NIHQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.147665980 (64h22m50.018749025s) [noise=true , address=145.79.227.25:4618 , piece_id=5NBPSN653OJAOTT5SQYUCHASNWGR7EFCFIAMUHYH3M7VILCUIAHQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.148271602 (64h22m50.019354637s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.149087727 (64h22m50.020170772s) [noise=true , address=45.91.116.5:4297 , piece_id=JYAFY55C5EYMVOYCF3LY25UKALYL76NR5SN5DKH3RB67WKGR62WQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.151023432 (64h22m50.022106477s) [noise=true , address=145.79.227.28:4727 , piece_id=XHE4AX4UFAA53UOTMI5TDTA2GQXM5QMWQDVKSGUGOQIXF6JA7PVA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:51.761342963 (64h22m51.632426008s) [upload=56436]: Next returned nil segment Dec 8 07:51:51.761356038 (64h22m51.632439083s) [upload=56436]: Waiting for error group managing segments... Dec 8 07:51:52.143112877 (64h22m52.014195912s) [noise=true , address=145.79.227.42:4122 , piece_id=5YUJVB6L5LYLRBJ2475TVR5UY7NWIWTOE7KLKQJAXKOZRTGXL5IQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.143778344 (64h22m52.014861389s) [noise=true , address=145.79.227.25:4618 , piece_id=5NBPSN653OJAOTT5SQYUCHASNWGR7EFCFIAMUHYH3M7VILCUIAHQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.146079326 (64h22m52.017162371s) [noise=true , address=145.79.227.20:4439 , piece_id=3AQ7WAXEFXGY6VGOQQACWGUGWDUDRAYIWXUQFBSZXWHFPVLMBIWQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.146479507 (64h22m52.017562542s) [noise=true , address=145.79.227.22:4510 , piece_id=MURUKN5W3JWOHKZGIDJ3NAF5TCHFOVM2A46QFIMLUP6JKQXGZIWQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.155737106 (64h22m52.026820151s) [noise=true , address=145.79.227.39:4015 , piece_id=4TBSZGVUKZRXFHSU3C2NSJ2H26ZKMMURPNJFEQGS2ICQI6KZUNKQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.161614518 (64h22m52.032697553s) [noise=true , address=145.79.227.23:4547 , piece_id=BCYPKLT75XINURN3XAXXBALSQOHSYB4PIUEVAOLI4OGBUSEPGKOA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.166235817 (64h22m52.037318862s) [noise=true , address=145.79.227.28:4727 , piece_id=XHE4AX4UFAA53UOTMI5TDTA2GQXM5QMWQDVKSGUGOQIXF6JA7PVA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.274077770 (64h22m52.145160815s) [noise=true , address=77.90.31.14:4663 , piece_id=KRLEX4OSLJJVKHVEOVAQALC374PIN3SK2L6BM63OVCAWZCS5NIHQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.311177544 (64h22m52.182260579s) [noise=true , address=45.91.116.5:4297 , piece_id=JYAFY55C5EYMVOYCF3LY25UKALYL76NR5SN5DKH3RB67WKGR62WQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.352570024 (64h22m52.223653069s) [noise=true , address=77.90.31.10:4519 , piece_id=BWYJYFKLJPOXDUYYNGS6OXZ4N3J62CRLSRKMIDBJNEUZB54IWEKQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.493961551 (64h22m52.365044586s) [noise=true , address=145.79.227.24:4582 , piece_id=ZVY2X2XMRLDJA74AL4UMYY5A6MCEO4L2GUI4G77CXHM54CL6XTCA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.494609204 (64h22m52.365692249s) [noise=true , address=145.79.227.44:4194 , piece_id=7HLBO2SUUG5ZFYGMTMWZ4XMN5VO4LQRVVTNQ46CZOBU3MB32EO6A, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.522301065 (64h22m52.3933841s) [noise=true , address=145.79.227.26:4655 , piece_id=KAK4VED4HFJXM6WO3ICYAUBWVHCQRKW4ACETAGAZYCHKJCLV6JRQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.522323438 (64h22m52.393406483s) [seg_pos={0 0}, upload=56436]: Segment reached optimal threshold of 13 pieces. Dec 8 07:51:52.522550460 (64h22m52.393633495s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 8 07:51:52.522558455 (64h22m52.3936415s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522657554 (64h22m52.393740599s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection Dec 8 07:51:52.522668415 (64h22m52.39375145s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522900286 (64h22m52.393983331s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: expected piece hash; context canceled; EOF Dec 8 07:51:52.522956193 (64h22m52.394039258s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522986951 (64h22m52.394069996s) [seg_pos={0 0}, upload=56436]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 07:51:52.523097702 (64h22m52.394180737s) [upload=56436]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 07:51:52.523136586 (64h22m52.394219631s) [upload=56436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 07:52:28.774033712 (64h23m28.645116757s) [upload=56436]: Done uploading segments... Dec 8 07:52:28.774096702 (64h23m28.645179747s) [upload=56436]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.406541365s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..................XX.........X...............X..............] duration: 5.640518186s trace: https://jaeger.dev.storj.io/trace/396bcc0ee1169960 FAIL (2025-12-08 08:24:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.640542212s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.187698547s trace: https://jaeger.dev.storj.io/trace/775323a46ba4bd95 FAIL (2025-12-08 08:24:25 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 6.18773666s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........................X........................X......X.] duration: 3.381986695s trace: https://jaeger.dev.storj.io/trace/5713ceca9b4db6c3 FAIL (2025-12-08 08:24:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.382025847s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 1.11769279s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/3eabec942a864841 FAIL (2025-12-08 08:24:20 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.117734198s core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X...........X...........................................] duration: 8.157347267s trace: https://jaeger.dev.storj.io/trace/6fbc0248558ad673 FAIL (2025-12-08 08:24:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 07:51:49.892373053 (64h22m49.763456099s) [upload=56436]: Starting upload Dec 8 07:51:49.892470558 (64h22m49.763553603s) [upload=56436]: Done starting upload Dec 8 07:51:49.892577121 (64h22m49.763660176s) [upload=56436]: Uploading segments... Dec 8 07:51:49.892739571 (64h22m49.763822606s) [upload=56436]: Got next segment. Inline: false Dec 8 07:51:49.892803042 (64h22m49.763886087s) [upload=56436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 07:51:49.892810626 (64h22m49.763893671s) [upload=56436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 07:51:50.139750947 (64h22m50.010833992s) [seg_pos={0 0}, upload=56436]: Begin upload segment... Dec 8 07:51:50.139806774 (64h22m50.010889819s) [seg_pos={0 0}, upload=56436]: Done begin upload segment. Dec 8 07:51:50.139840288 (64h22m50.010923333s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.139892166 (64h22m50.010975211s) [noise=true , address=145.79.227.24:4582 , piece_id=ZVY2X2XMRLDJA74AL4UMYY5A6MCEO4L2GUI4G77CXHM54CL6XTCA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.140602369 (64h22m50.011685414s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.141877286 (64h22m50.012960331s) [noise=true , address=145.79.227.23:4547 , piece_id=BCYPKLT75XINURN3XAXXBALSQOHSYB4PIUEVAOLI4OGBUSEPGKOA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.142886528 (64h22m50.013969563s) [noise=true , address=145.79.227.26:4655 , piece_id=KAK4VED4HFJXM6WO3ICYAUBWVHCQRKW4ACETAGAZYCHKJCLV6JRQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.143398483 (64h22m50.014481528s) [noise=true , address=145.79.227.42:4122 , piece_id=5YUJVB6L5LYLRBJ2475TVR5UY7NWIWTOE7KLKQJAXKOZRTGXL5IQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.143786251 (64h22m50.014869296s) [noise=true , address=145.79.227.22:4510 , piece_id=MURUKN5W3JWOHKZGIDJ3NAF5TCHFOVM2A46QFIMLUP6JKQXGZIWQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.144207654 (64h22m50.015290699s) [noise=true , address=77.90.31.10:4519 , piece_id=BWYJYFKLJPOXDUYYNGS6OXZ4N3J62CRLSRKMIDBJNEUZB54IWEKQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.144614459 (64h22m50.015697494s) [noise=true , address=145.79.227.20:4439 , piece_id=3AQ7WAXEFXGY6VGOQQACWGUGWDUDRAYIWXUQFBSZXWHFPVLMBIWQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.145229349 (64h22m50.016312394s) [noise=true , address=145.79.227.44:4194 , piece_id=7HLBO2SUUG5ZFYGMTMWZ4XMN5VO4LQRVVTNQ46CZOBU3MB32EO6A, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.146391893 (64h22m50.017474938s) [noise=true , address=145.79.227.39:4015 , piece_id=4TBSZGVUKZRXFHSU3C2NSJ2H26ZKMMURPNJFEQGS2ICQI6KZUNKQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.146973060 (64h22m50.018056105s) [noise=true , address=77.90.31.14:4663 , piece_id=KRLEX4OSLJJVKHVEOVAQALC374PIN3SK2L6BM63OVCAWZCS5NIHQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.147665980 (64h22m50.018749025s) [noise=true , address=145.79.227.25:4618 , piece_id=5NBPSN653OJAOTT5SQYUCHASNWGR7EFCFIAMUHYH3M7VILCUIAHQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.148271602 (64h22m50.019354637s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.149087727 (64h22m50.020170772s) [noise=true , address=45.91.116.5:4297 , piece_id=JYAFY55C5EYMVOYCF3LY25UKALYL76NR5SN5DKH3RB67WKGR62WQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.151023432 (64h22m50.022106477s) [noise=true , address=145.79.227.28:4727 , piece_id=XHE4AX4UFAA53UOTMI5TDTA2GQXM5QMWQDVKSGUGOQIXF6JA7PVA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:51.761342963 (64h22m51.632426008s) [upload=56436]: Next returned nil segment Dec 8 07:51:51.761356038 (64h22m51.632439083s) [upload=56436]: Waiting for error group managing segments... Dec 8 07:51:52.143112877 (64h22m52.014195912s) [noise=true , address=145.79.227.42:4122 , piece_id=5YUJVB6L5LYLRBJ2475TVR5UY7NWIWTOE7KLKQJAXKOZRTGXL5IQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.143778344 (64h22m52.014861389s) [noise=true , address=145.79.227.25:4618 , piece_id=5NBPSN653OJAOTT5SQYUCHASNWGR7EFCFIAMUHYH3M7VILCUIAHQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.146079326 (64h22m52.017162371s) [noise=true , address=145.79.227.20:4439 , piece_id=3AQ7WAXEFXGY6VGOQQACWGUGWDUDRAYIWXUQFBSZXWHFPVLMBIWQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.146479507 (64h22m52.017562542s) [noise=true , address=145.79.227.22:4510 , piece_id=MURUKN5W3JWOHKZGIDJ3NAF5TCHFOVM2A46QFIMLUP6JKQXGZIWQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.155737106 (64h22m52.026820151s) [noise=true , address=145.79.227.39:4015 , piece_id=4TBSZGVUKZRXFHSU3C2NSJ2H26ZKMMURPNJFEQGS2ICQI6KZUNKQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.161614518 (64h22m52.032697553s) [noise=true , address=145.79.227.23:4547 , piece_id=BCYPKLT75XINURN3XAXXBALSQOHSYB4PIUEVAOLI4OGBUSEPGKOA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.166235817 (64h22m52.037318862s) [noise=true , address=145.79.227.28:4727 , piece_id=XHE4AX4UFAA53UOTMI5TDTA2GQXM5QMWQDVKSGUGOQIXF6JA7PVA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.274077770 (64h22m52.145160815s) [noise=true , address=77.90.31.14:4663 , piece_id=KRLEX4OSLJJVKHVEOVAQALC374PIN3SK2L6BM63OVCAWZCS5NIHQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.311177544 (64h22m52.182260579s) [noise=true , address=45.91.116.5:4297 , piece_id=JYAFY55C5EYMVOYCF3LY25UKALYL76NR5SN5DKH3RB67WKGR62WQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.352570024 (64h22m52.223653069s) [noise=true , address=77.90.31.10:4519 , piece_id=BWYJYFKLJPOXDUYYNGS6OXZ4N3J62CRLSRKMIDBJNEUZB54IWEKQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.493961551 (64h22m52.365044586s) [noise=true , address=145.79.227.24:4582 , piece_id=ZVY2X2XMRLDJA74AL4UMYY5A6MCEO4L2GUI4G77CXHM54CL6XTCA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.494609204 (64h22m52.365692249s) [noise=true , address=145.79.227.44:4194 , piece_id=7HLBO2SUUG5ZFYGMTMWZ4XMN5VO4LQRVVTNQ46CZOBU3MB32EO6A, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.522301065 (64h22m52.3933841s) [noise=true , address=145.79.227.26:4655 , piece_id=KAK4VED4HFJXM6WO3ICYAUBWVHCQRKW4ACETAGAZYCHKJCLV6JRQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.522323438 (64h22m52.393406483s) [seg_pos={0 0}, upload=56436]: Segment reached optimal threshold of 13 pieces. Dec 8 07:51:52.522550460 (64h22m52.393633495s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 8 07:51:52.522558455 (64h22m52.3936415s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522657554 (64h22m52.393740599s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection Dec 8 07:51:52.522668415 (64h22m52.39375145s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522900286 (64h22m52.393983331s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: expected piece hash; context canceled; EOF Dec 8 07:51:52.522956193 (64h22m52.394039258s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522986951 (64h22m52.394069996s) [seg_pos={0 0}, upload=56436]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 07:51:52.523097702 (64h22m52.394180737s) [upload=56436]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 07:51:52.523136586 (64h22m52.394219631s) [upload=56436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 07:52:28.774033712 (64h23m28.645116757s) [upload=56436]: Done uploading segments... Dec 8 07:52:28.774096702 (64h23m28.645179747s) [upload=56436]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.157381603s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 4.883193618s trace: https://jaeger.dev.storj.io/trace/5699b43ed41f47e3 FAIL (2025-12-08 08:24:10 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.88322517s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X..X...........] duration: 2.304332913s trace: https://jaeger.dev.storj.io/trace/23e4fea4d1c3bcb8 FAIL (2025-12-08 08:24:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 08:16:58.813434894 (64h47m58.684517929s) [upload=56813]: Starting upload Dec 8 08:16:58.813552538 (64h47m58.684635583s) [upload=56813]: Done starting upload Dec 8 08:16:58.813656476 (64h47m58.684739521s) [upload=56813]: Uploading segments... Dec 8 08:16:58.813718364 (64h47m58.684801409s) [upload=56813]: Got next segment. Inline: true Dec 8 08:16:58.813724626 (64h47m58.684807661s) [upload=56813]: Waiting for error group managing segments... Dec 8 08:16:58.813831950 (64h47m58.684914995s) [upload=56813]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 08:16:58.813839474 (64h47m58.68492254s) [upload=56813]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 08:16:58.813843732 (64h47m58.684926777s) [upload=56813]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 08:17:25.151880477 (64h48m25.022963512s) [upload=56813]: Done uploading segments... Dec 8 08:17:25.151931593 (64h48m25.023014638s) [upload=56813]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.304362711s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.816110122s trace: https://jaeger.dev.storj.io/trace/775323a46ba4bd95 FAIL (2025-12-08 08:23:58 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.816137583s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...................XX.........X...............X.............] duration: 5.747407021s trace: https://jaeger.dev.storj.io/trace/396bcc0ee1169960 FAIL (2025-12-08 08:23:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.747441446s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X...........X..........................................] duration: 9.743508683s trace: https://jaeger.dev.storj.io/trace/6fbc0248558ad673 FAIL (2025-12-08 08:23:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 07:51:49.892373053 (64h22m49.763456099s) [upload=56436]: Starting upload Dec 8 07:51:49.892470558 (64h22m49.763553603s) [upload=56436]: Done starting upload Dec 8 07:51:49.892577121 (64h22m49.763660176s) [upload=56436]: Uploading segments... Dec 8 07:51:49.892739571 (64h22m49.763822606s) [upload=56436]: Got next segment. Inline: false Dec 8 07:51:49.892803042 (64h22m49.763886087s) [upload=56436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 07:51:49.892810626 (64h22m49.763893671s) [upload=56436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 07:51:50.139750947 (64h22m50.010833992s) [seg_pos={0 0}, upload=56436]: Begin upload segment... Dec 8 07:51:50.139806774 (64h22m50.010889819s) [seg_pos={0 0}, upload=56436]: Done begin upload segment. Dec 8 07:51:50.139840288 (64h22m50.010923333s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.139892166 (64h22m50.010975211s) [noise=true , address=145.79.227.24:4582 , piece_id=ZVY2X2XMRLDJA74AL4UMYY5A6MCEO4L2GUI4G77CXHM54CL6XTCA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.140602369 (64h22m50.011685414s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.141877286 (64h22m50.012960331s) [noise=true , address=145.79.227.23:4547 , piece_id=BCYPKLT75XINURN3XAXXBALSQOHSYB4PIUEVAOLI4OGBUSEPGKOA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.142886528 (64h22m50.013969563s) [noise=true , address=145.79.227.26:4655 , piece_id=KAK4VED4HFJXM6WO3ICYAUBWVHCQRKW4ACETAGAZYCHKJCLV6JRQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.143398483 (64h22m50.014481528s) [noise=true , address=145.79.227.42:4122 , piece_id=5YUJVB6L5LYLRBJ2475TVR5UY7NWIWTOE7KLKQJAXKOZRTGXL5IQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.143786251 (64h22m50.014869296s) [noise=true , address=145.79.227.22:4510 , piece_id=MURUKN5W3JWOHKZGIDJ3NAF5TCHFOVM2A46QFIMLUP6JKQXGZIWQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.144207654 (64h22m50.015290699s) [noise=true , address=77.90.31.10:4519 , piece_id=BWYJYFKLJPOXDUYYNGS6OXZ4N3J62CRLSRKMIDBJNEUZB54IWEKQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.144614459 (64h22m50.015697494s) [noise=true , address=145.79.227.20:4439 , piece_id=3AQ7WAXEFXGY6VGOQQACWGUGWDUDRAYIWXUQFBSZXWHFPVLMBIWQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.145229349 (64h22m50.016312394s) [noise=true , address=145.79.227.44:4194 , piece_id=7HLBO2SUUG5ZFYGMTMWZ4XMN5VO4LQRVVTNQ46CZOBU3MB32EO6A, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.146391893 (64h22m50.017474938s) [noise=true , address=145.79.227.39:4015 , piece_id=4TBSZGVUKZRXFHSU3C2NSJ2H26ZKMMURPNJFEQGS2ICQI6KZUNKQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.146973060 (64h22m50.018056105s) [noise=true , address=77.90.31.14:4663 , piece_id=KRLEX4OSLJJVKHVEOVAQALC374PIN3SK2L6BM63OVCAWZCS5NIHQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.147665980 (64h22m50.018749025s) [noise=true , address=145.79.227.25:4618 , piece_id=5NBPSN653OJAOTT5SQYUCHASNWGR7EFCFIAMUHYH3M7VILCUIAHQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.148271602 (64h22m50.019354637s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.149087727 (64h22m50.020170772s) [noise=true , address=45.91.116.5:4297 , piece_id=JYAFY55C5EYMVOYCF3LY25UKALYL76NR5SN5DKH3RB67WKGR62WQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.151023432 (64h22m50.022106477s) [noise=true , address=145.79.227.28:4727 , piece_id=XHE4AX4UFAA53UOTMI5TDTA2GQXM5QMWQDVKSGUGOQIXF6JA7PVA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:51.761342963 (64h22m51.632426008s) [upload=56436]: Next returned nil segment Dec 8 07:51:51.761356038 (64h22m51.632439083s) [upload=56436]: Waiting for error group managing segments... Dec 8 07:51:52.143112877 (64h22m52.014195912s) [noise=true , address=145.79.227.42:4122 , piece_id=5YUJVB6L5LYLRBJ2475TVR5UY7NWIWTOE7KLKQJAXKOZRTGXL5IQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.143778344 (64h22m52.014861389s) [noise=true , address=145.79.227.25:4618 , piece_id=5NBPSN653OJAOTT5SQYUCHASNWGR7EFCFIAMUHYH3M7VILCUIAHQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.146079326 (64h22m52.017162371s) [noise=true , address=145.79.227.20:4439 , piece_id=3AQ7WAXEFXGY6VGOQQACWGUGWDUDRAYIWXUQFBSZXWHFPVLMBIWQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.146479507 (64h22m52.017562542s) [noise=true , address=145.79.227.22:4510 , piece_id=MURUKN5W3JWOHKZGIDJ3NAF5TCHFOVM2A46QFIMLUP6JKQXGZIWQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.155737106 (64h22m52.026820151s) [noise=true , address=145.79.227.39:4015 , piece_id=4TBSZGVUKZRXFHSU3C2NSJ2H26ZKMMURPNJFEQGS2ICQI6KZUNKQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.161614518 (64h22m52.032697553s) [noise=true , address=145.79.227.23:4547 , piece_id=BCYPKLT75XINURN3XAXXBALSQOHSYB4PIUEVAOLI4OGBUSEPGKOA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.166235817 (64h22m52.037318862s) [noise=true , address=145.79.227.28:4727 , piece_id=XHE4AX4UFAA53UOTMI5TDTA2GQXM5QMWQDVKSGUGOQIXF6JA7PVA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.274077770 (64h22m52.145160815s) [noise=true , address=77.90.31.14:4663 , piece_id=KRLEX4OSLJJVKHVEOVAQALC374PIN3SK2L6BM63OVCAWZCS5NIHQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.311177544 (64h22m52.182260579s) [noise=true , address=45.91.116.5:4297 , piece_id=JYAFY55C5EYMVOYCF3LY25UKALYL76NR5SN5DKH3RB67WKGR62WQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.352570024 (64h22m52.223653069s) [noise=true , address=77.90.31.10:4519 , piece_id=BWYJYFKLJPOXDUYYNGS6OXZ4N3J62CRLSRKMIDBJNEUZB54IWEKQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.493961551 (64h22m52.365044586s) [noise=true , address=145.79.227.24:4582 , piece_id=ZVY2X2XMRLDJA74AL4UMYY5A6MCEO4L2GUI4G77CXHM54CL6XTCA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.494609204 (64h22m52.365692249s) [noise=true , address=145.79.227.44:4194 , piece_id=7HLBO2SUUG5ZFYGMTMWZ4XMN5VO4LQRVVTNQ46CZOBU3MB32EO6A, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.522301065 (64h22m52.3933841s) [noise=true , address=145.79.227.26:4655 , piece_id=KAK4VED4HFJXM6WO3ICYAUBWVHCQRKW4ACETAGAZYCHKJCLV6JRQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.522323438 (64h22m52.393406483s) [seg_pos={0 0}, upload=56436]: Segment reached optimal threshold of 13 pieces. Dec 8 07:51:52.522550460 (64h22m52.393633495s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 8 07:51:52.522558455 (64h22m52.3936415s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522657554 (64h22m52.393740599s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection Dec 8 07:51:52.522668415 (64h22m52.39375145s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522900286 (64h22m52.393983331s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: expected piece hash; context canceled; EOF Dec 8 07:51:52.522956193 (64h22m52.394039258s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522986951 (64h22m52.394069996s) [seg_pos={0 0}, upload=56436]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 07:51:52.523097702 (64h22m52.394180737s) [upload=56436]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 07:51:52.523136586 (64h22m52.394219631s) [upload=56436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 07:52:28.774033712 (64h23m28.645116757s) [upload=56436]: Done uploading segments... Dec 8 07:52:28.774096702 (64h23m28.645179747s) [upload=56436]: Upload finished. err: metaclient: context deadline exceeded Duration: 9.743533121s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........................X........................X......X] duration: 5.00105206s trace: https://jaeger.dev.storj.io/trace/5713ceca9b4db6c3 FAIL (2025-12-08 08:23:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001077307s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 740.51796ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/3eabec942a864841 FAIL (2025-12-08 08:23:35 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 740.544119ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X...........X.........................................] duration: 7.956512935s trace: https://jaeger.dev.storj.io/trace/6fbc0248558ad673 FAIL (2025-12-08 08:23:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 07:51:49.892373053 (64h22m49.763456099s) [upload=56436]: Starting upload Dec 8 07:51:49.892470558 (64h22m49.763553603s) [upload=56436]: Done starting upload Dec 8 07:51:49.892577121 (64h22m49.763660176s) [upload=56436]: Uploading segments... Dec 8 07:51:49.892739571 (64h22m49.763822606s) [upload=56436]: Got next segment. Inline: false Dec 8 07:51:49.892803042 (64h22m49.763886087s) [upload=56436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 07:51:49.892810626 (64h22m49.763893671s) [upload=56436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 07:51:50.139750947 (64h22m50.010833992s) [seg_pos={0 0}, upload=56436]: Begin upload segment... Dec 8 07:51:50.139806774 (64h22m50.010889819s) [seg_pos={0 0}, upload=56436]: Done begin upload segment. Dec 8 07:51:50.139840288 (64h22m50.010923333s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.139892166 (64h22m50.010975211s) [noise=true , address=145.79.227.24:4582 , piece_id=ZVY2X2XMRLDJA74AL4UMYY5A6MCEO4L2GUI4G77CXHM54CL6XTCA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.140602369 (64h22m50.011685414s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.141877286 (64h22m50.012960331s) [noise=true , address=145.79.227.23:4547 , piece_id=BCYPKLT75XINURN3XAXXBALSQOHSYB4PIUEVAOLI4OGBUSEPGKOA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.142886528 (64h22m50.013969563s) [noise=true , address=145.79.227.26:4655 , piece_id=KAK4VED4HFJXM6WO3ICYAUBWVHCQRKW4ACETAGAZYCHKJCLV6JRQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.143398483 (64h22m50.014481528s) [noise=true , address=145.79.227.42:4122 , piece_id=5YUJVB6L5LYLRBJ2475TVR5UY7NWIWTOE7KLKQJAXKOZRTGXL5IQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.143786251 (64h22m50.014869296s) [noise=true , address=145.79.227.22:4510 , piece_id=MURUKN5W3JWOHKZGIDJ3NAF5TCHFOVM2A46QFIMLUP6JKQXGZIWQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.144207654 (64h22m50.015290699s) [noise=true , address=77.90.31.10:4519 , piece_id=BWYJYFKLJPOXDUYYNGS6OXZ4N3J62CRLSRKMIDBJNEUZB54IWEKQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.144614459 (64h22m50.015697494s) [noise=true , address=145.79.227.20:4439 , piece_id=3AQ7WAXEFXGY6VGOQQACWGUGWDUDRAYIWXUQFBSZXWHFPVLMBIWQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.145229349 (64h22m50.016312394s) [noise=true , address=145.79.227.44:4194 , piece_id=7HLBO2SUUG5ZFYGMTMWZ4XMN5VO4LQRVVTNQ46CZOBU3MB32EO6A, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.146391893 (64h22m50.017474938s) [noise=true , address=145.79.227.39:4015 , piece_id=4TBSZGVUKZRXFHSU3C2NSJ2H26ZKMMURPNJFEQGS2ICQI6KZUNKQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.146973060 (64h22m50.018056105s) [noise=true , address=77.90.31.14:4663 , piece_id=KRLEX4OSLJJVKHVEOVAQALC374PIN3SK2L6BM63OVCAWZCS5NIHQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.147665980 (64h22m50.018749025s) [noise=true , address=145.79.227.25:4618 , piece_id=5NBPSN653OJAOTT5SQYUCHASNWGR7EFCFIAMUHYH3M7VILCUIAHQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.148271602 (64h22m50.019354637s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.149087727 (64h22m50.020170772s) [noise=true , address=45.91.116.5:4297 , piece_id=JYAFY55C5EYMVOYCF3LY25UKALYL76NR5SN5DKH3RB67WKGR62WQ, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:50.151023432 (64h22m50.022106477s) [noise=true , address=145.79.227.28:4727 , piece_id=XHE4AX4UFAA53UOTMI5TDTA2GQXM5QMWQDVKSGUGOQIXF6JA7PVA, seg_pos={0 0}, upload=56436]: Uploading piece... Dec 8 07:51:51.761342963 (64h22m51.632426008s) [upload=56436]: Next returned nil segment Dec 8 07:51:51.761356038 (64h22m51.632439083s) [upload=56436]: Waiting for error group managing segments... Dec 8 07:51:52.143112877 (64h22m52.014195912s) [noise=true , address=145.79.227.42:4122 , piece_id=5YUJVB6L5LYLRBJ2475TVR5UY7NWIWTOE7KLKQJAXKOZRTGXL5IQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.143778344 (64h22m52.014861389s) [noise=true , address=145.79.227.25:4618 , piece_id=5NBPSN653OJAOTT5SQYUCHASNWGR7EFCFIAMUHYH3M7VILCUIAHQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.146079326 (64h22m52.017162371s) [noise=true , address=145.79.227.20:4439 , piece_id=3AQ7WAXEFXGY6VGOQQACWGUGWDUDRAYIWXUQFBSZXWHFPVLMBIWQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.146479507 (64h22m52.017562542s) [noise=true , address=145.79.227.22:4510 , piece_id=MURUKN5W3JWOHKZGIDJ3NAF5TCHFOVM2A46QFIMLUP6JKQXGZIWQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.155737106 (64h22m52.026820151s) [noise=true , address=145.79.227.39:4015 , piece_id=4TBSZGVUKZRXFHSU3C2NSJ2H26ZKMMURPNJFEQGS2ICQI6KZUNKQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.161614518 (64h22m52.032697553s) [noise=true , address=145.79.227.23:4547 , piece_id=BCYPKLT75XINURN3XAXXBALSQOHSYB4PIUEVAOLI4OGBUSEPGKOA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.166235817 (64h22m52.037318862s) [noise=true , address=145.79.227.28:4727 , piece_id=XHE4AX4UFAA53UOTMI5TDTA2GQXM5QMWQDVKSGUGOQIXF6JA7PVA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.274077770 (64h22m52.145160815s) [noise=true , address=77.90.31.14:4663 , piece_id=KRLEX4OSLJJVKHVEOVAQALC374PIN3SK2L6BM63OVCAWZCS5NIHQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.311177544 (64h22m52.182260579s) [noise=true , address=45.91.116.5:4297 , piece_id=JYAFY55C5EYMVOYCF3LY25UKALYL76NR5SN5DKH3RB67WKGR62WQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.352570024 (64h22m52.223653069s) [noise=true , address=77.90.31.10:4519 , piece_id=BWYJYFKLJPOXDUYYNGS6OXZ4N3J62CRLSRKMIDBJNEUZB54IWEKQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.493961551 (64h22m52.365044586s) [noise=true , address=145.79.227.24:4582 , piece_id=ZVY2X2XMRLDJA74AL4UMYY5A6MCEO4L2GUI4G77CXHM54CL6XTCA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.494609204 (64h22m52.365692249s) [noise=true , address=145.79.227.44:4194 , piece_id=7HLBO2SUUG5ZFYGMTMWZ4XMN5VO4LQRVVTNQ46CZOBU3MB32EO6A, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.522301065 (64h22m52.3933841s) [noise=true , address=145.79.227.26:4655 , piece_id=KAK4VED4HFJXM6WO3ICYAUBWVHCQRKW4ACETAGAZYCHKJCLV6JRQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Dec 8 07:51:52.522323438 (64h22m52.393406483s) [seg_pos={0 0}, upload=56436]: Segment reached optimal threshold of 13 pieces. Dec 8 07:51:52.522550460 (64h22m52.393633495s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 8 07:51:52.522558455 (64h22m52.3936415s) [noise=true , address=145.79.227.29:4942 , piece_id=TOG6LFCQT55I7C7AS4BB7TSUBAOHWBYBADVAVD7RO72ZQ6KQFLYQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522657554 (64h22m52.393740599s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40528->77.90.31.12:4590: use of closed network connection Dec 8 07:51:52.522668415 (64h22m52.39375145s) [noise=true , address=77.90.31.12:4590 , piece_id=JVYF4PSPZWQCRSBZVTAHNNQ3IYARN4KERCMCYPA7KADOAGVSHECQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522900286 (64h22m52.393983331s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk): protocol: expected piece hash; context canceled; EOF Dec 8 07:51:52.522956193 (64h22m52.394039258s) [noise=true , address=145.79.227.21:4475 , piece_id=WH2XZJM5DYWHII55EKRD5F2UYCRD7M66TJ5OETEKKOJHNPEI27KQ, seg_pos={0 0}, upload=56436]: Long tail cancellation due to optimal threshold Dec 8 07:51:52.522986951 (64h22m52.394069996s) [seg_pos={0 0}, upload=56436]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 07:51:52.523097702 (64h22m52.394180737s) [upload=56436]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 07:51:52.523136586 (64h22m52.394219631s) [upload=56436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 07:52:28.774033712 (64h23m28.645116757s) [upload=56436]: Done uploading segments... Dec 8 07:52:28.774096702 (64h23m28.645179747s) [upload=56436]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.956540757s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 4.907076839s trace: https://jaeger.dev.storj.io/trace/5699b43ed41f47e3 FAIL (2025-12-08 08:23:16 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.907115673s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................X........................X......] duration: 3.511026451s trace: https://jaeger.dev.storj.io/trace/6e38355f6ad6bb63 FAIL (2025-12-08 08:23:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.511053053s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....................XX.........X...............X............] duration: 5.946335318s trace: https://jaeger.dev.storj.io/trace/396bcc0ee1169960 FAIL (2025-12-08 08:23:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.946365346s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.171611636s trace: https://jaeger.dev.storj.io/trace/775323a46ba4bd95 FAIL (2025-12-08 08:22:56 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.171641662s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X..X..........] duration: 2.163316582s trace: https://jaeger.dev.storj.io/trace/23e4fea4d1c3bcb8 FAIL (2025-12-08 08:22:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 08:16:58.813434894 (64h47m58.684517929s) [upload=56813]: Starting upload Dec 8 08:16:58.813552538 (64h47m58.684635583s) [upload=56813]: Done starting upload Dec 8 08:16:58.813656476 (64h47m58.684739521s) [upload=56813]: Uploading segments... Dec 8 08:16:58.813718364 (64h47m58.684801409s) [upload=56813]: Got next segment. Inline: true Dec 8 08:16:58.813724626 (64h47m58.684807661s) [upload=56813]: Waiting for error group managing segments... Dec 8 08:16:58.813831950 (64h47m58.684914995s) [upload=56813]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 08:16:58.813839474 (64h47m58.68492254s) [upload=56813]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 08:16:58.813843732 (64h47m58.684926777s) [upload=56813]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 08:17:25.151880477 (64h48m25.022963512s) [upload=56813]: Done uploading segments... Dec 8 08:17:25.151931593 (64h48m25.023014638s) [upload=56813]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.163342773s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)